Compare experiments

Compare experiment runs to decide whether a prompt, model, retrieval, or code change is ready to ship. Start with aggregate scores, then inspect cases that got worse and open their traces to understand the cause.

Experiment runs can use Langfuse datasets or local data. See Evaluate an existing application to create two comparable runs in Python or TypeScript, or explore the example project.

Choose comparable runs

Open Experiments, select the runs to compare, and open the comparison view. Choose the reviewed release run as the baseline.

For a release decision, use the same dataset version and evaluator definitions for baseline and candidate. Record the application commit, prompt or model version, and evaluator version in experiment metadata. A dataset version fixes the test data; it does not make model outputs deterministic.

You can compare runs from different data sources, but first check that the cases represent the same inputs and expected outputs. A missing case is not a passing case. For local data, preserve case identifiers in your test artifacts so your CI policy can match cases explicitly.

Inspect scores and outputs

Review score, cost, and latency differences to identify tradeoffs. An improvement in average quality can hide a regression on a critical case.

Both runs above have 50% accuracy. The average alone does not tell you whether the candidate is safe to release.

Use score thresholds in the comparison view to narrow the results, then inspect the baseline and candidate outputs side by side. Check the score explanation against the output. If an evaluator failed or returned no result, resolve that error before treating the comparison as complete.

Investigate and review failures

Open the trace for a failing item. Inspect the application output and the intermediate retrievals, model calls, or tools that produced it. For example, an incorrect refund window might come from outdated retrieved policy text or from the model ignoring the correct policy.

Separate application failures from evaluator mistakes. A good paraphrase can fail a string check; a fluent answer can still contain an unsupported claim. Use human scores to record the review outcome and failure reason. Use annotation queues when review needs to be shared across a team.

If you change the evaluator after review, re-score both versions with the same updated definition. Keep the original experiment metadata so the decision can be reproduced.

Turn the decision into a CI policy

Approve a baseline explicitly and record its identity with the dataset and evaluator versions. Do not automatically replace it with every successful candidate. A useful policy combines minimum aggregate scores with checks for newly failing critical cases and incomplete results.

See Experiments in CI/CD for an example. The baseline approval and gate policy live in your repository; choosing a baseline in the comparison UI does not configure a CI gate.

---

---