Replace Vibes With Evals: Build an Evaluation Pipeline

Most teams test an AI feature by typing in a few questions, deciding the answers look fine, and clicking deploy. Shipping on vibes works right up until it doesn't — a language model can be confidently wrong, and eyeballing ten answers tells you nothing about the thousands of conversations happening in production.

This walkthrough replaces vibes with something you can measure. Using a customer-support assistant for a fictional product called Nimbus Cloud, it builds a real evaluation pipeline around a live app: trace production, find the failures that matter, capture them into a dataset, score them automatically, and measure every change before it ships.

Start from production traces

The walkthrough assumes tracing is already set up, so every request shows up as a trace with the full story: what got retrieved from the docs, the prompt that was sent, the response, token counts, and latency — the whole pipeline in one view.

Monitoring does two jobs on top of that raw data: it tracks aggregate trends like cost, latency, and quality scores over time, and it surfaces the individual traces worth investigating right now.

Here's the shift that takes you from reactive to systematic: your production traffic is the best test set you'll ever have. It's full of messy, real-world questions you never would have thought to write yourself.

Error analysis: name what went wrong

Take one real failure. A user asked which regions Nimbus Cloud supports, and the agent confidently listed a region — eu-west-3 — that doesn't exist. It even picked up a thumbs-down.

That thumbs-down is explicit user feedback — the clearest signal you get. There's also implicit feedback, like a user rephrasing a question or abandoning the chat. Both land in Langfuse as scores.

Reading through traces like this and naming what went wrong is error analysis. For each failure, the real decision is whether it's a one-time prompt fix or a recurring pattern worth testing for. A hallucinated region is a pattern you never want to ship again — so capture it.

Capture failures into a dataset

A dataset is a collection of test cases you run your app against every time you make a change. Create one from the sidebar, then, whenever you find a trace where the app got something wrong, open it and add it to the dataset.

Langfuse pulls in the original input automatically. You fill in the expected output — the answer you actually wanted, in this case the four real regions and nothing invented. That's a reference answer: a gold-standard response the judge can compare against later.

Do this a few times and the dataset becomes a living benchmark of the exact cases your application needs to get right — one that mirrors real production traffic. That's how a failure stops being an incident and becomes a permanent regression test.

Score automatically with evaluators

Grading thousands of answers by hand doesn't scale, so you need automated judges. There are three ways to evaluate, and you'll use all of them:

Reading outputs yourself — where you always start, to build intuition for what "good" means.
Code evaluators — deterministic checks like valid JSON or a length limit.
LLM-as-a-judge — for anything that needs understanding language, like whether an answer is actually correct.

To set up an LLM judge, pick the dimension to measure (here, a managed evaluator for correctness) and the model that does the judging. Langfuse gives you the scoring rubric as an editable prompt. Then decide what it runs on: point it at live observations so production answers get scored automatically as they happen — sampling, say, 10% of traffic to keep costs sane — or run it against a controlled dataset in an experiment.

Either way, the key step is variable mapping: you tell the judge which field is the question, which is the answer your app produced, and which is the correct answer to compare against. Because it compares against a known correct answer, this is a reference-based evaluator. If a value is buried in a nested JSON payload, point at it with a JSONPath expression; a live preview shows the fully filled-in prompt so you know the mapping is right.

One caveat: an LLM judge isn't perfect. Calibrate it against a few human-labeled examples so you trust it's measuring what you think, and prefer simple pass/fail scores over a vague one-to-five scale.

Measure every change with an experiment

To grade against the corrected answers in your dataset, run the evaluator as an experiment. Every experiment has the same anatomy: a baseline (your current production setup), the dataset, one variable you're changing, and the outputs you compare.

Change just the prompt and keep everything else fixed. From the dataset, run the experiment: pick the new prompt, choose the dataset, attach the correctness evaluator, and map the variables. Langfuse runs the new version across every dataset item and scores each answer.

Then compare both runs side by side. Read a few traces manually first — that's how you learn what "better" actually means for your app — and let the scores make it concrete.

In the video, the new prompt lifts average correctness from around 0.65 to 0.90. But one row regresses: the "What is the default query timeout?" question now answers 60 seconds instead of the correct 30. On vibes, that regression ships straight to users. With an experiment, you catch it before it does.

The full loop

That's the whole cycle:

Trace what happens in production.
Use error analysis to find the failures that matter.
Capture them into a dataset.
Score them automatically with evaluators.
Run experiments so every change is measured before it ships.

Production then feeds the cycle all over again, and your agent gets measurably better with real usage instead of drifting on guesswork. All of it is open source, and trace data is stored in ClickHouse under the hood, so it scales to billions of traces. You can also wire experiments into CI/CD to catch regressions on every pull request.

To go deeper on any single step:

Evaluation overview — all evaluation methods in Langfuse
Datasets and experiments
LLM-as-a-judge evaluators and code evaluators
Experiments in CI/CD — run evals alongside your pull requests
Metrics and monitoring — track cost, latency, and quality over time
Langfuse Academy — a deeper course on evaluation

Was this page helpful?