---
title: Build a human-in-the-loop scoring workflow
sidebarTitle: Human-in-the-loop scoring
description: Capture human judgments from your own review tooling and ingest them into Langfuse as structured scores, validated against a score config.
category: Evaluation
---

# Build a human-in-the-loop scoring workflow

Many teams keep a human in the loop to judge LLM output. Langfuse offers built-in [annotation queues](/docs/evaluation/evaluation-methods/annotation-queues) for this, but sometimes you already have your own internal review tool, or want reviewers to work inside an existing app.

This guide shows how to connect that custom tooling to Langfuse: pull the traces your reviewers should look at, then ingest their judgments back as [scores](/docs/evaluation/scores/overview) via the SDK or API. Standardizing those scores against a [score config](/faq/all/manage-score-configs) keeps them consistent for later analysis.

  If you do not need to own the review UI, use Langfuse [annotation
  queues](/docs/evaluation/evaluation-methods/annotation-queues) instead. They
  give you a built-in reviewer interface, queue management, and score configs
  without writing any tooling. If you are considering building your own UI
  because annotation queues are missing a feature you need, we would love to
  hear about it: please open a [feature
  request](https://github.com/orgs/langfuse/discussions/new?category=ideas) on
  GitHub.

## Prerequisites

- Traces already flowing into Langfuse from your application. See [tracing](/docs/observability/get-started) if you have not set this up yet.
- A custom annotation or review UI where your reviewers view traces and submit their judgments.
- The Langfuse SDK installed and credentials (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`) available in your environment.

## Walkthrough

<Steps>

### Define a score config

A [score config](/faq/all/manage-score-configs) standardizes the schema your reviewers score against, so every reviewer submits the same shape of data. This config needs to be created in Langfuse first.

_For example, for a support QA tool you might define a categorical config named `support_quality` with the categories `excellent`, `acceptable`, and `poor`._

See [how to create and manage score configs](/faq/all/manage-score-configs) to set this up, then note the resulting `configId`, which you will reference when ingesting scores.

### Surface traces to your reviewers

Pull the traces your reviewers should look at, then render their inputs and outputs in your tool. You can do this via the SDKs or the [public API](/docs/api-and-data-platform/features/public-api), filtering by name, user, tags, or time range to build a review queue.

<LangTabs items={["Python SDK", "JS/TS SDK", "API"]}>
<Tab>

```python
from langfuse import get_client
from datetime import datetime, timedelta, timezone
import json

langfuse = get_client()

now = datetime.now(timezone.utc)

# List one root observation per trace for review, e.g. all traces of a given name.
# The advanced `filter` parameter takes precedence over the individual query
# parameters, so the time window is expressed as filter conditions as well.
filters = json.dumps([
    {"type": "boolean", "column": "isRootObservation", "operator": "=", "value": True},
    {"type": "string", "column": "traceName", "operator": "=", "value": "support-conversation"},
    {"type": "datetime", "column": "startTime", "operator": ">=", "value": (now - timedelta(days=7)).isoformat()},
    {"type": "datetime", "column": "startTime", "operator": "<", "value": now.isoformat()},
])

response = langfuse.api.observations.get_many(
    filter=filters,
    fields="core,basic,io,trace_context",
    limit=50,
)

for root_observation in response.data:
    # Render the root observation's input / output in your review tool
    print(root_observation.trace_id, root_observation.input)
```

_See the [Python SDK reference](https://python.reference.langfuse.com)._

</Tab>
<Tab>

```ts
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();

const now = new Date();
const sevenDaysAgo = new Date(now.getTime() - 7 * 24 * 60 * 60 * 1000);

// List one root observation per trace for review, e.g. all traces of a given name.
// The advanced `filter` parameter takes precedence over the individual query
// parameters, so the time window is expressed as filter conditions as well.
const filter = JSON.stringify([
  { type: "boolean", column: "isRootObservation", operator: "=", value: true },
  { type: "string", column: "traceName", operator: "=", value: "support-conversation" },
  { type: "datetime", column: "startTime", operator: ">=", value: sevenDaysAgo.toISOString() },
  { type: "datetime", column: "startTime", operator: "<", value: now.toISOString() },
]);

const response = await langfuse.api.observations.getMany({
  filter,
  fields: "core,basic,io,trace_context",
  limit: 50,
});

for (const rootObservation of response.data) {
  // Render the root observation's input / output in your review tool
  console.log(rootObservation.traceId, rootObservation.input);
}
```

_See the [JS/TS SDK reference](https://js.reference.langfuse.com)._

</Tab>
<Tab>

```bash
curl -G https://cloud.langfuse.com/api/public/v2/observations \
  -u "pk-lf-...":"sk-lf-..." \
  --data-urlencode 'filter=[{"type":"boolean","column":"isRootObservation","operator":"=","value":true},{"type":"string","column":"traceName","operator":"=","value":"support-conversation"},{"type":"datetime","column":"startTime","operator":">=","value":"2026-08-01T00:00:00Z"}]' \
  --data-urlencode "fields=core,basic,io" \
  --data-urlencode "limit=50"
```

The v2 API returns observation rows, not trace objects. Filtering on `isRootObservation` gives you one row per trace; render each row's input and output, and fetch all observations for a `traceId` when reviewers need the full trace. Because `filter` takes precedence over the individual query parameters, keep every condition — including the time window — inside the filter array. See [Migration of deprecated APIs](/faq/all/deprecated-api-migration#traces) for details.

_See the [API reference](https://api.reference.langfuse.com/#tag/observationsv2/GET/api/public/v2/observations)._

</Tab>
</LangTabs>

  All three tabs use the supported observations-first read path. Query logical
  root observations (`isRootObservation = true`) when you need one review row
  per trace, and fetch all observations for a given `traceId` when your
  reviewers need the full trace activity. If you only need the root filter and a
  time window, use the first-class `isRootObservation`, `fromStartTime`, and
  `toStartTime` query parameters instead of `filter`. See [Migration of
  deprecated APIs](/faq/all/deprecated-api-migration#traces) for the mapping
  from older trace reads.

### Ingest the reviewer's judgment as a score

When a reviewer submits their decision, write it back to Langfuse as a score, referencing the `configId` from step 1 so the value is validated against the schema.

Attach the score to whatever the reviewer is judging: a single response (`trace_id`, and optionally `observation_id` to target a specific step within the trace), or a full [session](/docs/observability/features/sessions) (`session_id`) when they rate a conversation as a whole.

<LangTabs items={["Python SDK", "JS/TS SDK", "API"]}>
<Tab>

```python
from langfuse import get_client

langfuse = get_client()

# Score a single response
langfuse.create_score(
    trace_id="trace_id_here",
    name="support_quality",
    value="acceptable",  # must match a category in the config
    data_type="CATEGORICAL",
    config_id="your_config_id",  # validates the value against the config
    comment="Resolved the issue but tone was a bit terse.",
)

# Or score a full conversation by passing session_id instead of trace_id
langfuse.create_score(
    session_id="session_id_here",
    name="support_quality",
    value="excellent",
    data_type="CATEGORICAL",
    config_id="your_config_id",
)
```

_See the [Python SDK reference](https://python.reference.langfuse.com)._

</Tab>
<Tab>

```ts
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();

// Score a single response
langfuse.score.create({
  traceId: "trace_id_here",
  name: "support_quality",
  value: "acceptable", // must match a category in the config
  dataType: "CATEGORICAL",
  configId: "your_config_id", // validates the value against the config
  comment: "Resolved the issue but tone was a bit terse.",
});

// Or score a full conversation by passing sessionId instead of traceId
langfuse.score.create({
  sessionId: "session_id_here",
  name: "support_quality",
  value: "excellent",
  dataType: "CATEGORICAL",
  configId: "your_config_id",
});

// Flush the scores in short-lived environments
await langfuse.flush();
```

_See the [JS/TS SDK reference](https://js.reference.langfuse.com)._

</Tab>
<Tab>

```bash
curl -X POST https://cloud.langfuse.com/api/public/scores \
  -u "pk-lf-...":"sk-lf-..." \
  -H "Content-Type: application/json" \
  -d '{
    "traceId": "trace_id_here",
    "name": "support_quality",
    "value": "acceptable",
    "dataType": "CATEGORICAL",
    "configId": "your_config_id",
    "comment": "Resolved the issue but tone was a bit terse."
  }'
```

_See the [API reference](https://api.reference.langfuse.com)._

</Tab>
</LangTabs>

When you reference a config, Langfuse validates the score before storing it: the name must match the config, a categorical value must map to one of the configured categories, and a numeric value must fall within the configured range. See [enforcing a score config](/docs/evaluation/evaluation-methods/scores-via-sdk#enforcing-a-score-config) for the full validation rules.

</Steps>

## What you can do with the scores

Once reviews are flowing in, the scores appear on the linked traces and sessions in the Langfuse UI. You can filter and slice on them, fetch them via the public API to drive your own dashboards, or use [score analytics](/docs/evaluation/scores/score-analytics) to track reviewer agreement and quality trends over time.

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/guides/human-in-the-loop-scoring.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.