---
title: Langfuse Evaluator Library
description: "Introducing the Langfuse Evaluator Library with prebuilt evaluators. Plus, enjoy a revamped UX with trace and variable previews for easier LLM evaluation."
date: 2025-05-24
author: Marlies
badge: Launch Week 3 🚀
ogImage: /images/changelog/2025-05-24-langfuse-evaluator-library.png
showOgInHeader: false
canonical: /docs/evaluation/evaluation-methods/llm-as-a-judge
---
> **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) and the API/SDK reference (https://api.reference.langfuse.com).
On Day 6 of our [Launch Week #3](/blog/2025-05-19-launch-week-3), we're introducing the Langfuse Evaluator Library and major improvements to the evaluator UX.
Evaluation is core to monitor and continuously improve LLM applications. There are many ways to eval LLM applications/agents and you can flexibly record these evals as [scores](/docs/scores/overview) in Langfuse.
## Langfuse Evaluator Library
The Langfuse LLM-as-a-Judge runner is all about making it easy to:
1. manage evaluation templates,
2. using your own models,
3. defining when these evals shall be run by being able to filter and sample your production and development data, and
4. interactively working with the results to improve your application.
Today, we introduce a larger library of prebuilt evaluators in partnership with [RAGAS](https://docs.ragas.io/en/stable/) to measure context relevance, SQL semantic equivalence, hallucinations, and other key dimensions.
While you can bring your own evaluation templates, the expanded library makes it easier to get started.
## Revamped Evaluator UX
Langfuse LLM-as-a-Judge is flexible (see above) but also a bit complex to configure. Thus, we introduced some core UX changes to make it easier to get started:
### 1. Standard Eval Model
A "standard eval model" can be configured that applies to all evals (unless you override it).
### 2. Preview of traces that match filter conditions
Langfuse allows you to filter traces by various conditions. We now show a preview of historical traces that match the filter conditions.
### 3. Preview of inserted variables
When you insert variables into your eval, we now show a preview of the variables that will be inserted.
## Getting Started
To get started, check out the [LLM-as-a-Judge Docs](/docs/scores/model-based-evals) or the walkthrough above.
Do you have any feedback? Please let us know via [GitHub](/issues)!
---
## Agent Instructions
This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2025-05-24-langfuse-evaluator-library.md`.
### Querying these docs
If the answer is not on this page, query the documentation instead of guessing:
- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:
```bash
curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
```
- **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).
### Before writing Langfuse code
- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal.
Found an error in these docs? Please open an issue at .