---
date: 2025-09-17
title: Experiment Runner SDK
description: New high-level SDK abstraction for running experiments on datasets with automatic tracing, concurrent execution, and flexible evaluation.
author: Hassieb
ogImage: /images/changelog/2025-09-12-experiment-sdk.png
canonical: /docs/evaluation/experiments/experiments-via-sdk
---

> **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) and the API/SDK reference (https://api.reference.langfuse.com).

Both the Python and JS/TS SDKs now provide a high-level abstraction for running experiments on datasets. The dataset can be both local or hosted on Langfuse. Using the Experiment runner is the recommended way to run an experiment on a dataset with our SDK.

**Key Features**

The experiment runner automatically handles:

- **Concurrent execution** of tasks with configurable limits
- **Automatic tracing** of all executions for observability
- **Flexible evaluation** with both item-level and run-level evaluators
- **Error isolation** so individual failures don't stop the experiment
- **Traces in Langfuse** even though the core task function is not instrumented by automatic input / return value capture
- **Dataset integration** for easy comparison and tracking

**Example**

<LangTabs items={["Python SDK", "JS/TS SDK"]}>
<Tab>

```python
from langfuse import get_client
from langfuse.openai import OpenAI

# Initialize client
langfuse = get_client()

# Define your task function
def my_task(*, item, **kwargs):
    question = item["input"]

    response = OpenAI().chat.completions.create(
        model="gpt-4.1", messages=[{"role": "user", "content": question}]
    )

    return response.choices[0].message.content

# Run experiment on local data
local_data = [
    {"input": "What is the capital of France?"},
    {"input": "What is the capital of Germany?"},
]

result = langfuse.run_experiment(
    name="Geography Quiz",
    description="Testing basic functionality",
    data=local_data,
    task=my_task,
)

# Pretty print results
print(result.format())
```

This prints:

```
1. Item 1:
   Input:    What is the capital of France?
   Actual:   The capital of France is Paris.

   Trace ID: e52488cb13d426f55a2a7c178d4cb0d0

2. Item 2:
   Input:    What is the capital of Germany?
   Actual:   The capital of Germany is **Berlin**.

   Trace ID: 188cd8fc165446fa957a7c15423cbe0e

──────────────────────────────────────────────────
📊 Geography Quiz - Testing basic functionality
2 items
```

</Tab>
<Tab>

```typescript
import { OpenAI } from "openai";
import { NodeSDK } from "@opentelemetry/sdk-node";
import {
  LangfuseClient,
  ExperimentTask,
  ExperimentItem,
} from "@langfuse/client";
import { observeOpenAI } from "@langfuse/openai";
import { LangfuseSpanProcessor } from "@langfuse/otel";

// Initialize OpenTelemetry
const otelSdk = new NodeSDK({ spanProcessors: [new LangfuseSpanProcessor()] });
otelSdk.start();

// Initialize client
const langfuse = new LangfuseClient();

// Define your task function
const myTask: ExperimentTask = async (item) => {
  const question = item.input;

  const response = await observeOpenAI(new OpenAI()).chat.completions.create({
    model: "gpt-4.1",
    messages: [{ role: "user", content: question }],
  });

  return response.choices[0].message.content;
};

// Run experiment on local data
const localData: ExperimentItem[] = [
  { input: "What is the capital of France?" },
  { input: "What is the capital of Germany?" },
];

const result = await langfuse.experiment.run({
  name: "Geography Quiz",
  description: "Testing basic functionality",
  data: localData,
  task: myTask,
});

console.log(await result.format());

// Important: shut down OpenTelemetry to ensure traces are sent to Langfuse
await otelSdk.shutdown();
```

This prints:

```
1. Item 1:
   Input:    What is the capital of France?
   Expected: null
   Actual:   The capital of France is **Paris**.

   Trace:
   https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/f8e12f19b4114621106512b923a1170f

2. Item 2:
   Input:    What is the capital of Germany?
   Expected: null
   Actual:   The capital of Germany is **Berlin**.

   Trace:
   https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/9852ca4665ceea9f83c2acfccf1b4052

──────────────────────────────────────────────────
📊 Geography Quiz - Testing basic functionality
2 items
```

</Tab>
</LangTabs>

**Get Started**

Learn more about the experiment runner incl. how to use it with Langfuse datasets, adding evaluators and more in our [remote dataset runs documentation](/docs/evaluation/dataset-runs/remote-run#experiment-runner-sdk).

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2025-09-17-experiment-runner-sdk.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.