Why is my observation-level evaluator not executing?

You've attached an LLM-as-a-Judge evaluator to an observation-level rule, but no scores appear. The rule may not match incoming observations, the evaluator mapping may not fit the rule's data, or the evaluator may be blocked by its LLM connection or model configuration.

There are a couple of things you can check:

1. Are you on a compatible SDK version or ingestion method?
2. If using trace-level filters: are you propagating attributes to observations?
3. Do your rule filters match actual observation data?
4. Do all mapped variables exist on matching observations?
5. Is your evaluator blocked by its LLM connection or model configuration?

Incompatible SDK version or ingestion method

Observation-level evaluators only work with data ingested via the OTEL endpoint. This means you need either:

• An OTel-based SDK: Python v3+ or JS/TS v4+ (these use the OTEL endpoint automatically)
• Direct OTEL ingestion: Sending OpenTelemetry spans to Langfuse's /api/public/otel endpoint

Data sent via the legacy REST ingestion API (/api/public/ingestion) or legacy SDKs (Python v2, JS/TS v3) does not produce observations in the format required for observation-level evaluation.

How to check your SDK version:

pip show langfuse

npm list langfuse

If you're using a custom ingestion pipeline (not an SDK), you need to send data to the OTEL endpoint instead of the legacy ingestion endpoint. Follow the custom ingestion migration guide to convert legacy events and emit v4-ready observations.

Trace-level attributes not propagated to observations

When your evaluator uses trace-level filters like tags, userId, sessionId, or metadata, the evaluator checks these attributes on the observation itself, it does not look up the parent trace. If you only set these attributes on the trace (e.g., via update_current_trace() in Python SDK v3 / updateActiveTrace() in JS SDK v4 or earlier), the observations won't have them, and the evaluator won't match.

Solution: Use propagate_attributes() (Python) or propagateAttributes() (JS/TS) to copy trace-level attributes to all observations created within a scope.

from langfuse import get_client, propagate_attributes

langfuse = get_client()

with langfuse.start_as_current_observation(as_type="span", name="user-workflow"):
    with propagate_attributes(
        user_id="user_123",
        session_id="session_abc",
        tags=["online_evaluator:my-eval"],
        metadata={"team": "support"},
    ):
        # All observations created inside this block
        # inherit the propagated attributes
        with langfuse.start_as_current_observation(
            as_type="generation", name="llm-call"
        ):
            pass

import { startActiveObservation, propagateAttributes } from "@langfuse/tracing";

await startActiveObservation("user-workflow", async () => {
  await propagateAttributes(
    {
      userId: "user_123",
      sessionId: "session_abc",
      tags: ["online_evaluator:my-eval"],
      metadata: { team: "support" },
    },
    async () => {
      // All observations created inside this callback
      // inherit the propagated attributes
    }
  );
});

Call propagate_attributes() early in your trace, before creating the observations you want to evaluate. Only attributes propagated this way will be available for filter matching on observations. See the instrumentation guide for more details.

Filter configuration mismatch

Your rule filters might not match what's actually on the observations. Because there is no error when nothing matches, this can be hard to spot.

Common mismatches:

• Observation name: The name must exactly match what your instrumentation produces. Go to a trace in the Langfuse UI, click on the observation you want to evaluate, and check its name.
• Observation type: Make sure you're filtering for the right type (GENERATION, SPAN, or EVENT). An LLM call is typically a GENERATION, while a wrapper function is usually a SPAN.
• Tag values: Tags are matched as exact strings. If your evaluator filters for my-eval but your observation has online_evaluator:my-eval, they won't match.
• Metadata values: Similar to tags, metadata keys and values must match exactly.

How to check: Open the rule and inspect the matching observations shown with its filters. If it shows matches but evaluations still do not run, check the other causes on this page, such as SDK version, attribute propagation, variable mappings, or the LLM connection.

Variable mapping references missing data

All variable mappings are required. An evaluator has default mappings, and a rule assignment can override them. If an observation matches the rule but a mapped field does not exist (for example, you map a variable to observation.metadata.tool_call and that field is absent), the evaluator errors instead of producing a score.

How to check: Open the evaluator and test it with an observation matched by the rule. Confirm that the prompt variables contain the expected data and correct any mapping errors before saving.

How to fix:

• Make sure the field exists on every observation that matches your filters
• If only some observations have the field, tighten your filters (e.g., add an observation name filter) to exclude observations that are missing it
• Consider mapping variables to fields that are always present, like observation.input or observation.output

LLM connection

If observations are matching but scores still are not appearing, the evaluator may be blocked by the LLM connection or model configuration.

How to check: Open the evaluator and look for a blocked status or banner. Then go to Settings → LLM Connections and verify:

• The API key is valid and not expired
• The model supports structured output (required for parsing evaluation results)

After fixing the connection or model configuration, reactivate the evaluator. See LLM Connections for configuration details.

Still stuck? Reach out to support.