Evaluate with Datasets

This guide walks you through how to set up a dataset, run an experiment on it, and evaluate the results. This is useful for evaluating changes before you deploy them to production. If you don't yet know what to evaluate, Choosing what to evaluate will help you further. For how datasets and experiments fit together, see Datasets and Experiments.

This walkthrough is for running experiments via SDK: you fetch a dataset from Langfuse, run your agent externally and push the results to the Langfuse platform. You can also run prompt experiments in the UI, score traces in the UI, or review items in an annotation queue.

Agentic installation

Install the Langfuse Agent Skill to let your coding agent access all Langfuse features.

Ask your coding agent to install the skill by pointing to the GitHub repository and instruct it to get started with offline evaluation.

Agent instruction

Install the Langfuse Agent Skill from github.com/langfuse/skills
and use it to create a first dataset for this application
with Langfuse.

Langfuse has a Cursor Plugin that includes the skill automatically.

Then prompt your agent:

Agent instruction

Set up offline evaluation for this application with Langfuse.

Install via npm (skills CLI):

npx skills add langfuse/skills --skill "langfuse"

If you want to target a specific agent directly:

npx skills add langfuse/skills --skill "langfuse" --agent "<agent-id>"

Alternatively you can manually clone the skill

Clone repo somewhere stable

git clone https://github.com/langfuse/skills.git /path/to/langfuse-skills

Make sure your agent's skills dir exists

mkdir -p /path/to/<agent-skill-root>/skills

Symlink the skill folder

ln -s /path/to/langfuse-skills/skills/langfuse /path/to/<agent-skill-root>/skills/langfuse

Then prompt your agent:

Agent instruction

Set up offline evaluation for this application with Langfuse.

Manual setup

Running experiments to test the performance of your system has three aspects:

Dataset: Test cases with inputs and expected outputs
Experiment condition: The application variation or model call you want to test
Evaluators: Functions that score the output

This guide uses the current Langfuse SDKs: Python SDK v4 and JS/TS SDK v5. Both use Langfuse's OpenTelemetry-based tracing and the current experiment runner. If you use an older SDK, see the Python v3 → v4 or JS/TS v4 → v5 migration guide.

Create a project and get API keys

Create a Langfuse account or self-host Langfuse.
Create a project and open Settings → API Keys.
Create an API key for your model provider. This example uses OpenAI.

Set the keys as environment variables:

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"
export OPENAI_API_KEY="sk-..."

Use the base URL for your Langfuse Cloud data region or self-hosted deployment.

Install the SDKs

pip install langfuse openai

# pnpm
pnpm add @langfuse/client @langfuse/openai @langfuse/otel @opentelemetry/sdk-node openai tsx

# npm
npm install @langfuse/client @langfuse/openai @langfuse/otel @opentelemetry/sdk-node openai tsx

Create the dataset

A dataset is a collection of test cases. Each item has an input and, optionally, an expected output that evaluators can compare against. A first dataset might have a handful of representative cases; later you can grow it as you learn what fails.

In the example below we add five questions about San Francisco tourist sites.

Run this setup script once; the experiment script in the next step reuses the stored dataset.

seed_dataset.py

from langfuse import get_client

langfuse = get_client()
dataset_name = "san-francisco-sites"

langfuse.create_dataset(
    name=dataset_name,
    description="Questions and expected answers about sites in San Francisco",
)

items = [
    {
        "id": "evaluation-quickstart-sf-golden-gate-bridge",
        "input": {
            "question": "Which red-orange suspension bridge connects San Francisco with Marin County?"
        },
        "expected_output": "Golden Gate Bridge",
    },
    {
        "id": "evaluation-quickstart-sf-alcatraz-island",
        "input": {
            "question": "Which island in San Francisco Bay is home to a former federal prison?"
        },
        "expected_output": "Alcatraz Island",
    },
    {
        "id": "evaluation-quickstart-sf-palace-of-fine-arts",
        "input": {
            "question": "Which Beaux-Arts landmark in the Marina District features a rotunda beside a lagoon?"
        },
        "expected_output": "Palace of Fine Arts",
    },
    {
        "id": "evaluation-quickstart-sf-coit-tower",
        "input": {
            "question": "Which Art Deco tower stands on Telegraph Hill?"
        },
        "expected_output": "Coit Tower",
    },
    {
        "id": "evaluation-quickstart-sf-lombard-street",
        "input": {
            "question": "Which San Francisco street is famous for a steep block with eight hairpin turns?"
        },
        "expected_output": "Lombard Street",
    },
]

for item in items:
    langfuse.create_dataset_item(dataset_name=dataset_name, **item)

print(f"Created {dataset_name} with {len(items)} items")

Run the setup script:

python seed_dataset.py

Run this setup script once; the experiment script in the next step reuses the stored dataset.

seed-dataset.ts

import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();
const datasetName = "san-francisco-sites";

const items = [
  {
    id: "evaluation-quickstart-sf-golden-gate-bridge",
    input: {
      question:
        "Which red-orange suspension bridge connects San Francisco with Marin County?",
    },
    expectedOutput: "Golden Gate Bridge",
  },
  {
    id: "evaluation-quickstart-sf-alcatraz-island",
    input: {
      question:
        "Which island in San Francisco Bay is home to a former federal prison?",
    },
    expectedOutput: "Alcatraz Island",
  },
  {
    id: "evaluation-quickstart-sf-palace-of-fine-arts",
    input: {
      question:
        "Which Beaux-Arts landmark in the Marina District features a rotunda beside a lagoon?",
    },
    expectedOutput: "Palace of Fine Arts",
  },
  {
    id: "evaluation-quickstart-sf-coit-tower",
    input: {
      question: "Which Art Deco tower stands on Telegraph Hill?",
    },
    expectedOutput: "Coit Tower",
  },
  {
    id: "evaluation-quickstart-sf-lombard-street",
    input: {
      question:
        "Which San Francisco street is famous for a steep block with eight hairpin turns?",
    },
    expectedOutput: "Lombard Street",
  },
];

async function main() {
  await langfuse.api.datasets.create({
    name: datasetName,
    description: "Questions and expected answers about sites in San Francisco",
  });

  for (const item of items) {
    await langfuse.dataset.createItem({
      datasetName,
      ...item,
    });
  }

  console.log(`Created ${datasetName} with ${items.length} items`);
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

Run the setup script:

npx tsx seed-dataset.ts

In the Langfuse UI, go to Datasets and click + New dataset. Name it san-francisco-sites if you want to run the example experiment in the next step without changing the script.

Add items one by one, import a CSV, or add cases from existing traces. For the San Francisco example, each item's input is a question and the expected output is the site name. Use any number of items — and any input shape — that matches your own use case.

See Datasets for CSV import, adding items from traces, and dataset versions.

When you create items via the SDK, dataset item IDs are stable, so adding an item with the same ID updates it instead of creating a duplicate.

Run an experiment

The example uses the SDK to run an experiment. This lets you run your agent in its own harness and access to tools outside of Langfuse. If you only need to test a prompt + model combination, you can also run prompt experiments in the UI.

You can still start an SDK experiment from the UI by configuring a webhook to trigger external execution.

In the example below, each dataset item is sent to the model. The evaluator returns 1 when the output exactly matches the expected site name and 0 otherwise.

sf_sites.py

from langfuse import Evaluation, get_client
from langfuse.openai import OpenAI

langfuse = get_client()
client = OpenAI()


def answer_question(*, item, **kwargs):
    question = item.input["question"]
    response = client.responses.create(
        model="gpt-5-mini",
        input=[
            {
                "role": "system",
                "content": "Answer the question about a site in San Francisco.",
            },
            {"role": "user", "content": question},
        ],
    )
    return response.output_text


def exact_match(*, output, expected_output, **kwargs):
    return Evaluation(
        name="exact_match",
        value=1.0 if output == expected_output else 0.0,
    )


try:
    dataset = langfuse.get_dataset("san-francisco-sites")
    result = dataset.run_experiment(
        name="San Francisco sites",
        run_name="San Francisco sites v1",
        description="First prompt for answering questions about San Francisco sites",
        task=answer_question,
        evaluators=[exact_match],
    )

    print(result.format())
finally:
    # Required for short-lived processes so all traces reach Langfuse.
    langfuse.flush()

Run the experiment:

python sf_sites.py

sf-sites.ts

import { LangfuseClient, type Evaluator } from "@langfuse/client";
import { observeOpenAI } from "@langfuse/openai";
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
import OpenAI from "openai";

const otelSdk = new NodeSDK({
  spanProcessors: [new LangfuseSpanProcessor()],
});
otelSdk.start();

const langfuse = new LangfuseClient();
const client = observeOpenAI(new OpenAI());

const exactMatch: Evaluator = async ({ output, expectedOutput }) => ({
  name: "exact_match",
  value: output === expectedOutput ? 1 : 0,
});

async function main() {
  const dataset = await langfuse.dataset.get("san-francisco-sites");

  const result = await dataset.runExperiment({
    name: "San Francisco sites",
    runName: "San Francisco sites v1",
    description:
      "First prompt for answering questions about San Francisco sites",
    task: async (item) => {
      const { question } = item.input as { question: string };
      const response = await client.responses.create({
        model: "gpt-5-mini",
        input: [
          {
            role: "system",
            content: "Answer the question about a site in San Francisco.",
          },
          { role: "user", content: question },
        ],
      });

      return response.output_text;
    },
    evaluators: [exactMatch],
  });

  console.log(await result.format({ includeItemResults: true }));
}

main()
  .catch((error) => {
    console.error(error);
    process.exitCode = 1;
  })
  // Required for short-lived processes so all traces reach Langfuse.
  .finally(() => otelSdk.shutdown());

Run the experiment:

npx tsx sf-sites.ts

The experiment runner handles concurrent execution, traces each task, attaches evaluator scores, and creates a dataset run that you can inspect and compare in Langfuse.

View the results

The formatted terminal output includes the experiment summary and a link to the dataset run. You can also open Experiments in Langfuse.

For each item, you can inspect:

The dataset input and expected output
The model's response
The exact-match score
The traced OpenAI call, including latency, token usage, and cost

Improve the prompt and compare

Exact match is intentionally strict. An answer such as The answer is Coit Tower. receives 0 because it does not exactly equal Coit Tower.

Change the run name to San Francisco sites v2 and replace the system message with:

Answer the question about a site in San Francisco. Return only the site's official name, with no additional text or punctuation.

Run the script again. Both runs use the same stored dataset, so you can compare aggregate scores and individual outputs side by side in the Experiments view.

FAQ

What if the dataset already exists?

The setup script creates the dataset and is intended to run once. If san-francisco-sites already exists, skip the dataset creation call and keep the item-creation loop. Stable item IDs make those calls safe to repeat.

Why is the experiment missing traces?

Check your Langfuse environment variables. For Python, keep langfuse.flush() at the end of the script. For JS/TS, keep otelSdk.shutdown() so the short-lived process exports all buffered spans before it exits.

Why are some exact-match scores zero?

Open the affected item and compare the model output with the expected output. Capitalization, extra words, and punctuation all cause exact match to fail. This is useful for deterministic output contracts; use LLM-as-a-Judge when correctness requires semantic judgment.

Next steps

Learn how datasets and dataset versions support reproducible tests.
Compare prompts and models without writing a task function using experiments via UI.
Explore the full experiment runner SDK, including async evaluators, concurrency, and run-level metrics.
Add experiments to CI/CD to catch regressions before deployment.
Write evaluators you can trust.
Choose between code evaluators, LLM-as-a-Judge, and human review for your application.

Was this page helpful?

On this page