--- date: 2026-04-15 title: Manage LLM-as-a-Judge evaluators via the API description: Create, version, and update LLM-as-a-Judge evaluators and evaluation rules programmatically through the (unstable) public API. author: Marlies canonical: /docs/evaluation/evaluation-methods/llm-as-a-judge --- > **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) and the API/SDK reference (https://api.reference.langfuse.com). A new [stable API for evaluators and evaluation rules](/changelog/2026-08-27-stable-evaluator-api) is now available under `/api/public/v2`. The unstable API described below remains available until November 16, 2026 (2026-11-16). You can now set up and manage LLM-as-a-Judge evaluation programmatically through the public API, in addition to the Langfuse UI. This lets you define how data is scored and what data gets evaluated entirely in code, so your evaluation setup can live in version control and roll out the same way across projects. Common things this unlocks: - **Version-control your evaluators** — keep judge prompts, output definitions, and model configuration in your repo and create new versions through CI. - **Replicate setups across projects** — script the same evaluators and rules into staging and production instead of recreating them by hand. - **Automate rollouts** — enable, pause, or repoint evaluation rules as part of a deployment pipeline. The endpoints are designed to be explored and consumed by coding agents. Point an agent at the API reference and let it create evaluators and wire up evaluation rules for you, rather than clicking through the UI. The API splits the setup into two resources: - **Evaluators** define _how_ to score data: the judge prompt, its `{{variables}}`, the structured output definition (numeric, boolean, or categorical), and the optional model configuration. Evaluators are versioned — creating one under an existing name produces the next version, and active rules automatically move to it. - **Evaluation rules** define _what_ gets evaluated: the target (live observations or experiments), filters, sampling rate, and the mapping from your data onto the evaluator's variables. Each rule references an evaluator family by `name` and `scope`. ``` POST /api/public/unstable/evaluators GET /api/public/unstable/evaluators GET /api/public/unstable/evaluators/{evaluatorId} POST /api/public/unstable/evaluation-rules GET /api/public/unstable/evaluation-rules GET /api/public/unstable/evaluation-rules/{evaluationRuleId} PUT /api/public/unstable/evaluation-rules/{evaluationRuleId} DELETE /api/public/unstable/evaluation-rules/{evaluationRuleId} ``` These endpoints are **unstable** and may change while the underlying evaluation data model is being redesigned. The UI workflow remains fully supported. ## Get started - [LLM-as-a-Judge documentation](/docs/evaluation/evaluation-methods/llm-as-a-judge) - [Evaluators API reference](https://api.reference.langfuse.com/#tag/unstableevaluators) --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2026-04-15-llm-as-a-judge-api.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .