--- title: Evaluation sidebarTitle: Evaluation description: How to judge whether experiment outputs are good, using manual review, code evaluator checks, and LLM-as-a-judge evaluators. --- # Evaluation ## How evaluation fits into the loop Offline evaluation is the step in the loop between running an experiment and shipping a change. You have a dataset, you have run your application against it, and now you need to judge whether the outputs are good. The AI Engineering Loop: - [Trace](/academy/tracing): traces, sessions, agents, prompts - [Monitor](/academy/monitoring): dashboards, LLM-as-judge, feedback - [Build datasets](/academy/datasets): datasets, features-as-tests - [Experiment](/academy/experiments): prompts, models, code variants - [Evaluate](/academy/evaluate): judges, custom evals, annotation ## How evaluation typically evolves [#how-evaluation-typically-evolves] Most of the time, you start by **manually reviewing outputs** to build intuition for what good and bad look like in your application. From there, you **identify specific failure modes** worth checking for. Once you can define them precisely, you **automate with dedicated evaluators**. The rest of this page covers the different kinds of evaluation in detail. In practice, you'll likely end up combining all of them. But the path to a well-functioning automated evaluation setup almost always starts from manual review. Manual evaluation is not a one-and-done step. Good production setups incorporate continuous review by human experts to catch new failure modes and keep automated evaluators calibrated. [Keeping the set alive](/academy/evaluate/choosing-what-to-evaluate#keeping-the-set-alive) covers this cadence. > **Guide: [Error analysis](/guides/cookbook/error-analysis-llm-applications)** > > Select your sample data, build an annotation queue, cluster failure categories, quantify failure rates, and decide what to do. ## Evaluation methods [#three-kinds-of-evaluation] There are three main ways to evaluate: manually, with code, or with an LLM. Each is suited to different kinds of quality checks. ### Manual evaluation Manual evaluation is the process of manually looking at outputs and scoring it/writing down your thoughts on its quality. This is an important process, reading outputs builds an understanding of what your application actually does, where it struggles, and what "good" looks like for your specific use case. That understanding is what tells you which automated evaluators to build and how to define their criteria later on. Teams that skip this step and jump straight to automated evaluation often end up measuring things that don't matter. Manual evaluation also produces human labels that serve as ground truth for validating automated evaluators later. ### Code evaluators [Code evaluators](/docs/evaluation/evaluation-methods/code-evaluators) check properties that can be verified with deterministic logic. In Langfuse, they are UI-authored Python or TypeScript evaluators that run on observations or experiments, not trace-level targets. They are fast, cheap, and produce the same result every time. Some example checks where code evaluators are a natural fit: - The output is valid JSON or follows a required schema - The output contains (or does not contain) specific keywords or patterns - The output stays within a length limit - The generated SQL is syntactically valid Their limitation is that they cannot assess meaning. A code evaluator can check that an output contains the word "refund," but it cannot check whether the output correctly explains the refund policy. ### LLM-as-a-judge An [LLM-as-a-judge evaluator](/docs/evaluation/evaluation-methods/llm-as-a-judge) uses a language model to score outputs. It is required to overcome the core issue that quality of AI Applications/Agents depends on grading the quality of a text output. This is the right method for qualities that require understanding language: whether a response is relevant to the question, whether the tone matches the intended audience, whether a summary captures the key points of the source material, etc. LLM judges are imperfect and easy to get wrong. This means: - A model does not automatically grade things as a human expert would as they do not have the context of the expert - They need [calibration](/guides/llm-as-a-judge-calibration-skill) against human preferences to verify they are measuring what you think they are measuring - They can share blind spots with your application's LLM, especially when the same model family is used for both These limitations aren't reasons to avoid LLM judges. An LLM judge that has been calibrated against human labels and is backed by code evaluator checks is a reliable evaluator. How to write one is covered in our deeper dive into [Writing good evaluators](/academy/evaluate/writing-evaluators). ## Reference-based vs reference-free evaluators [#reference-based-vs-reference-free] Both code evaluators and LLM-as-a-judge evaluators can be either reference-based or reference-free. A reference-based evaluator compares the output against a predefined expected output, like a correct answer or a golden response. A reference-free evaluator assesses the output on its own, without needing a ground truth to compare against. | | Code evaluators | LLM-as-a-judge / manual | | --------------- | ------------------------ | ----------------------------------------------------------- | | Reference-based | Exact string match check | "Does the response correctly explain this specific policy?" | | Reference-free | Validate JSON structure | Tone-of-voice evaluation; language detection | The advantage of reference-free evaluators is that they can be applied to unseen production data, while reference-based evaluators always need a pre-defined reference response. ## In practice ### When to set up evaluators [#when-to-set-up-evaluators] [As mentioned before](#how-evaluation-typically-evolves), you always start by manually reviewing. Once you have done that, not every failure you find needs an evaluator: a **one-time fix** you can resolve with a prompt change doesn't, a **generalization problem** you need to catch repeatedly does. [Choosing what to evaluate](/academy/evaluate/choosing-what-to-evaluate#fix-first) covers this further. ### What should you evaluate? Generic qualities like "helpfulness" or "quality" are tempting starting points, but they rarely produce useful signal. An evaluator that checks a vague criterion will give vague results. The more precisely you can define what "good" or "bad" looks like for your application, the more useful your evaluators will be. This page on [choosing what to evaluate](/academy/evaluate/choosing-what-to-evaluate) explores this further. When you do design an evaluator, prefer binary (pass/fail) scores over graded scales (1-5). [Writing good evaluators](/academy/evaluate/writing-evaluators#binary-verdicts) explains why. ### Combining evaluation methods Each quality you care about gets [its own evaluator](/academy/evaluate/writing-evaluators#one-evaluator-per-failure-mode). **Most mature evaluation setups use [all three evaluation methods](#three-kinds-of-evaluation).** Together, they give you a view on overall quality of your application. To make this concrete, here are three example applications and their evaluator stacks: _A SQL agent that converts natural-language questions into SQL against a company's data warehouse. Users ask things like "what was our revenue by region last quarter?" and the agent generates and runs the query._ | Quality | Method | | ---------------------------------- | -------------- | | Query is syntactically valid | Code evaluator | | Query answers the user's intent | LLM-as-a-judge | | Calibration on a continuous sample | Manual review | _A tone-of-voice rewriter that takes draft marketing copy and rewrites it to match the company's brand voice. The output is plain prose — no structure to check, no exact reference answer._ | Quality | Method | | ------------------------------------ | -------------- | | Tone matches the brand voice | LLM-as-a-judge | | Original message captured faithfully | LLM-as-a-judge | | Calibration on a continuous sample | Manual review | _An invoice-processing tool that extracts structured fields (vendor, total, due date) from uploaded PDFs. The dataset has exact-match labels for each field._ | Quality | Method | | ------------------------------------- | --------------------------------------------------- | | Extracted JSON matches the schema | Code evaluator | | Each field matches the expected value | Code evaluator (exact match against dataset labels) | | Edge cases and new vendor formats | Manual review (sample) | ## Where to start Start with manual review, then automate only the checks you need to run repeatedly. 1. [Review outputs manually](#how-evaluation-typically-evolves) to build intuition for what good and bad look like in your application. 2. Write down the specific failure modes you want to catch and define them as clearly as possible. 3. Set up an automated evaluator only when you need to test that failure mode repeatedly across many inputs or over time. The deeper dive on [writing good evaluators](/academy/evaluate/writing-evaluators) covers how to build one you can trust. When you are ready to score live production traces in Langfuse, follow [Evaluate Production Traffic](/docs/evaluation/get-started/online). ## What comes next If the results are good enough, you can ship the change. Once it is live, the loop starts again: the updated system produces new [traces](/academy/tracing), new [monitoring](/academy/monitoring) signals, and new opportunities to improve. Some evaluators should also move beyond offline experiments. Reference-free evaluators, user feedback signals, and other production-safe checks can be applied to live traffic to confirm that quality in production matches what you saw before deployment. If production behavior matches expectations, you can keep scaling with more confidence. If it does not, capture those cases in traces, turn them into [dataset](/academy/datasets) items, and run the next round of [experiments](/academy/experiments). That is how you close the loop. ## More guides - [LLM-as-a-judge calibration](/guides/llm-as-a-judge-calibration-skill) — Test whether your judge prompt agrees with how you would label cases, and iterate until it does. - [RAG evaluation with RAGAS](/resources/engineering/evaluation-of-rag-with-ragas) — Score retrieval and generation quality for a RAG application. - [Writing scores to Langfuse from external evaluation pipelines](/guides/cookbook/example_external_evaluation_pipelines) — Fetch production traces, evaluate them outside Langfuse, and post scores back via the API. - [Evaluating multi-turn conversations](/guides/cookbook/example_evaluating_multi_turn_conversations) — Score quality across conversation turns in chat applications. - [Simulated multi-turn conversations](/guides/cookbook/example_simulated_multi_turn_conversations) — Generate and evaluate simulated multi-turn scenarios. --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/academy/evaluate.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .