---
title: A/B Testing
sidebarTitle: A/B Testing
description: Use Open Source Prompt Management in Langfuse to systematically test and improve your LLM prompts with A/B testing.
---

# A/B Testing of LLM Prompts

[Langfuse Prompt Management](/docs/prompts/get-started) enables A/B testing by allowing you to label different versions of a prompt (e.g., `prod-a` and `prod-b`). Your application can randomly alternate between these versions, while Langfuse tracks performance metrics like response latency, cost, token usage, and evaluation metrics for each version.

**When to use A/B testing?**

A/B testing helps you see how different prompt versions work in real situations, adding to what you learn from testing on datasets. This works best when:

- Your app has good ways to measure success, deals with many different kinds of user inputs, and can handle some ups and downs in performance. This usually works for consumer apps where mistakes aren't a big deal.
- You've already tested thoroughly on your test data and want to try your changes with a small group of users before rolling out to everyone (also called canary deployment).

## Implementation

<Steps>

### Label your Prompt Versions

Label your prompt versions (e.g., `prod-a` and `prod-b`) to identify different variants for testing.

### Fetch Prompts and Run A/B Test

<LangTabs items={["Python SDK", "JS/TS SDK"]}>
<Tab>

```python
from langfuse import get_client
import random
from langfuse.openai import openai

# Requires environment variables for initialization
from langfuse import get_client
langfuse = get_client()

# Fetch prompt versions
prompt_a = langfuse.get_prompt("my-prompt-name", label="prod-a")
prompt_b = langfuse.get_prompt("my-prompt-name", label="prod-b")

# Randomly select version
selected_prompt = random.choice([prompt_a, prompt_b])

# Use in LLM call
response = openai.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": selected_prompt.compile(variable="value")}],
    # Link prompt to generation for analytics
    langfuse_prompt=selected_prompt
)
result_text = response.choices[0].message.content
```

</Tab>
<Tab>

```js
import { LangfuseClient } from "@langfuse/client";
import { observeOpenAI } from "@langfuse/openai";

import OpenAI from "openai";

// Requires environment variables for initialization
const langfuse = new LangfuseClient();

// Create and wrap OpenAI client
const openai = observeOpenAI(new OpenAI());

// Fetch prompt versions
const promptA = await langfuse.prompt.get("my-prompt-name", {
  label: "prod-a",
});
const promptB = await langfuse.prompt.get("my-prompt-name", {
  label: "prod-b",
});

// Randomly select version
const selectedPrompt = Math.random() < 0.5 ? promptA : promptB;

// Use in LLM call
const completion = await openai.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [
    {
      role: "user",
      content: selectedPrompt.compile({ variable: "value" }),
    },
  ],
  // Link prompt to generation for analytics
  langfusePrompt: selectedPrompt,
});
const resultText = completion.choices[0].message.content;
```

</Tab>
</LangTabs>

Refer to [prompt management documentation](/docs/prompts/get-started) for additional examples on how to fetch and use prompts.

### Analyze Results

Compare metrics for each prompt version in the Langfuse UI:

**Key metrics available for comparison:**

- Response latency and token usage
- Cost per request
- Quality evaluation scores
- Custom metrics you define

</Steps>

## Related Resources

- To benchmark complete application behavior on datasets (not just prompt selection), use [Experiments](/docs/evaluation/core-concepts#experiments).

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/docs/prompt-management/features/a-b-testing.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.