---
title: Why is my observation-level evaluator not executing?
description: If you set up an observation-level LLM-as-a-Judge evaluator but no scores appear, here's how to diagnose and fix it.
tags: [evaluation, evaluation-get-started]
---

# Why is my observation-level evaluator not executing?

You've attached an [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge) evaluator to an observation-level rule, but no scores appear. The rule may not match incoming observations, the evaluator mapping may not fit the rule's data, or the evaluator may be blocked by its LLM connection or model configuration.

There are a couple of things you can check:

1. [Are you on a compatible SDK version or ingestion method?](#incompatible-sdk-version-or-ingestion-method)
2. [If using trace-level filters: are you propagating attributes to observations?](#trace-level-attributes-not-propagated-to-observations)
3. [Do your rule filters match actual observation data?](#filter-configuration-mismatch)
4. [Do all mapped variables exist on matching observations?](#variable-mapping-references-missing-data)
5. [Is your evaluator blocked by its LLM connection or model configuration?](#llm-connection)

## Incompatible SDK version or ingestion method

Observation-level evaluators only work with data ingested via the [OTEL endpoint](/integrations/native/opentelemetry). This means you need either:

- An **OTel-based SDK**: Python v3+ or JS/TS v4+ (these use the OTEL endpoint automatically)
- **Direct OTEL ingestion**: Sending OpenTelemetry spans to Langfuse's `/api/public/otel` endpoint

Data sent via the **legacy REST ingestion API** (`/api/public/ingestion`) or **legacy SDKs** (Python v2, JS/TS v3) does not produce observations in the format required for observation-level evaluation.

**How to check your SDK version:**

<LangTabs items={["Python", "JS/TS"]}>
<Tab>

```bash
pip show langfuse
```

You need version `3.0.0` or higher. If you're on v2, follow the [Python v2 → v3 migration guide](/docs/observability/sdk/upgrade-path/python-v2-to-v3).

</Tab>
<Tab>

```bash
npm list langfuse
```

You need version `4.0.0` or higher. If you're on v3, follow the [JS/TS v3 → v4 migration guide](/docs/observability/sdk/upgrade-path/js-v3-to-v4).

</Tab>
</LangTabs>

**If you're using a custom ingestion pipeline** (not an SDK), you need to send data to the OTEL endpoint instead of the legacy ingestion endpoint. Follow the [custom ingestion migration guide](/integrations/native/opentelemetry/migration-to-v4) to convert legacy events and emit v4-ready observations.

## Trace-level attributes not propagated to observations

When your evaluator uses trace-level filters like `tags`, `userId`, `sessionId`, or `metadata`, the evaluator checks these attributes **on the observation itself**, it does not look up the parent trace. If you only set these attributes on the trace (e.g., via `update_current_trace()` in Python SDK v3 / `updateActiveTrace()` in JS SDK v4 or earlier), the observations won't have them, and the evaluator won't match.

**Solution**: Use `propagate_attributes()` (Python) or `propagateAttributes()` (JS/TS) to copy trace-level attributes to all observations created within a scope.

<LangTabs items={["Python", "JS/TS"]}>
<Tab>

```python
from langfuse import get_client, propagate_attributes

langfuse = get_client()

with langfuse.start_as_current_observation(as_type="span", name="user-workflow"):
    with propagate_attributes(
        user_id="user_123",
        session_id="session_abc",
        tags=["online_evaluator:my-eval"],
        metadata={"team": "support"},
    ):
        # All observations created inside this block
        # inherit the propagated attributes
        with langfuse.start_as_current_observation(
            as_type="generation", name="llm-call"
        ):
            pass
```

</Tab>
<Tab>

```typescript
import { startActiveObservation, propagateAttributes } from "@langfuse/tracing";

await startActiveObservation("user-workflow", async () => {
  await propagateAttributes(
    {
      userId: "user_123",
      sessionId: "session_abc",
      tags: ["online_evaluator:my-eval"],
      metadata: { team: "support" },
    },
    async () => {
      // All observations created inside this callback
      // inherit the propagated attributes
    }
  );
});
```

</Tab>
</LangTabs>

Call `propagate_attributes()` early in your trace, before creating the observations you want to evaluate. Only attributes propagated this way will be available for filter matching on observations. See the [instrumentation guide](/docs/observability/sdk/instrumentation#add-attributes) for more details.

## Filter configuration mismatch

Your rule filters might not match what's actually on the observations. Because there is no error when nothing matches, this can be hard to spot.

**Common mismatches:**

- **Observation name**: The name must exactly match what your instrumentation produces. Go to a trace in the Langfuse UI, click on the observation you want to evaluate, and check its name.
- **Observation type**: Make sure you're filtering for the right type (`GENERATION`, `SPAN`, or `EVENT`). An LLM call is typically a `GENERATION`, while a wrapper function is usually a `SPAN`.
- **Tag values**: Tags are matched as exact strings. If your evaluator filters for `my-eval` but your observation has `online_evaluator:my-eval`, they won't match.
- **Metadata values**: Similar to tags, metadata keys and values must match exactly.

**How to check**: Open the rule and inspect the matching observations shown with its filters. If it shows matches but evaluations still do not run, check the other causes on this page, such as SDK version, attribute propagation, variable mappings, or the LLM connection.

## Variable mapping references missing data

All variable mappings are **required**. An evaluator has default mappings, and a rule assignment can override them. If an observation matches the rule but a mapped field does not exist (for example, you map a variable to `observation.metadata.tool_call` and that field is absent), the evaluator errors instead of producing a score.

**How to check**: Open the evaluator and test it with an observation matched by the rule. Confirm that the prompt variables contain the expected data and correct any mapping errors before saving.

**How to fix:**

- Make sure the field exists on every observation that matches your filters
- If only some observations have the field, tighten your filters (e.g., add an observation name filter) to exclude observations that are missing it
- Consider mapping variables to fields that are always present, like `observation.input` or `observation.output`

## LLM connection

If observations are matching but scores still are not appearing, the evaluator may be blocked by the LLM connection or model configuration.

**How to check**: Open the evaluator and look for a blocked status or banner. Then go to **Settings → LLM Connections** and verify:

- The API key is valid and not expired
- The model supports structured output (required for parsing evaluation results)

After fixing the connection or model configuration, reactivate the evaluator. See [LLM Connections](/docs/administration/llm-connection) for configuration details.

Still stuck? Reach out to [support](/support).

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/faq/all/observation-eval-not-executing.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.