LLM-as-a-Judge
LLM-as-a-judge is a technique to evaluate the quality of LLM applications by using an LLM as a judge. The LLM is given a trace or a dataset entry and asked to score and reason about the output. The scores and reasoning are stored as scores in Langfuse.
Why use LLM-as-a-judge?
- Scalable & cost‑effective: Judge thousands of outputs quickly and cheaply versus human panels.
- Human‑like judgments: Captures nuance (helpfulness, safety, coherence) better than simple metrics, especially when rubric‑guided.
- Repeatable comparisons: With a fixed rubric, you can rerun the same prompts to get consistent scores and short rationales.
Set up step-by-step
Create a new LLM-as-a-Judge evaluator
Navigate to the Evaluators page and click on the + Set up Evaluator button.
Set the default model
Next, you’ll define the default model used for conducting the evaluations. The default is used by every managed evaluator; custom templates may override it.
This step requires an LLM Connection to be set up. Please see LLM Connections for more information.
- Setup: This default model needs to be set up once, though it can be changed at any point if needed.
- Change: Existing evaluators keep evaluating with the new model—historic results stay preserved.
- Structured Output Support: It’s crucial that the chosen default model supports structured output. This is essential for our system to correctly interpret the evaluation results from the LLM judge.
Pick an Evaluator
Now we select an evaluator. There are two main ways:
Langfuse ships a growing catalog of evaluators built and maintained by us and partners like Ragas. Each evaluator captures best-practice evaluation prompts for a specific quality dimension—e.g. Hallucination, Context-Relevance, Toxicity, Helpfulness.
- Ready to use: no prompt writing required.
- Continuously expanded: by adding OSS partner-maintained evaluators and more evaluator types in the future (e.g. regex-based).
Choose which Data to Evaluate
With your evaluator and model selected, you now specify which data to run the evaluations on. You can chose between running on production tracing data or Datasets during Dataset Runs.
Evaluating live production traffic allows you to monitor the performance of your LLM application in real-time.
- Scope: Choose whether to run on new traces only and/or existing traces once (for backfilling). When in doubt, we recommend running on new traces.
- Filter: Narrow down the evaluation to a specific subset of data you’re interested in. You can filter by trace name, tags,
userIdand may more. Combine filters freely. - Preview: Langfuse shows a sample of traces from the last 24 hours that match your current filters, allowing you to sanity-check your selection.
- Sampling: To manage costs and evaluation throughput, you can configure the evaluator to run on a percentage (e.g., 5%) of the matched traces.
Map Variables & preview Evaluation Prompt
You now need to teach Langfuse which properties of your trace or dataset item represent the actual data to populate these variables for a sensible evaluation. For instance, you might map your system’s logged trace input to the prompt’s {{input}} variable, and the LLM response ie trace output to the prompt’s {{output}} variable. This mapping is crucial for ensuring the evaluation is sensible and relevant.
- Prompt Preview: As you configure the mapping, Langfuse shows a live preview of the evaluation prompt populated with actual data. This preview uses historical traces from the last 24 hours that matched your filters (from Step 3). You can navigate through several example traces to see how their respective data fills the prompt, helping you build confidence that the mapping is correct.
- JSONPath: If the data is nested (e.g., within a JSON object), you can use a JSONPath expression (like
$.choices[0].message.content) to precisely locate it.
✨ Done! You have successfully set up an evaluator which will run on your data.
Need custom logic? Use the SDK instead—see Custom Scores or an external pipeline example.
Monitor & Iterate
As our system evaluates your data it writes the results as scores. You can then:
- View Logs: Check detailed logs for each evaluation, including status, any retry errors, and the full request/response bodies sent to the evaluation model.
- Use Dashboards: Aggregate scores over time, filter by version or environment, and track the performance of your LLM application.
- Take Actions: Pause, resume, or delete an evaluator.
