--- date: 2026-03-20 title: Categorical LLM-as-a-Judge Scores description: LLM-as-a-Judge evaluators can now return categorical scores in addition to numeric ones. ogImage: /images/changelog/2026-03-20-categorical-llm-as-a-judge-scores.jpg author: Hassieb canonical: /docs/evaluation/evaluation-methods/llm-as-a-judge --- > **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) and the API/SDK reference (https://api.reference.langfuse.com). LLM-as-a-Judge evaluators in Langfuse can now return categorical scores in addition to numeric ones. You can define a fixed set of allowed categories in the evaluator template, have the judge choose from them, and store the result as a native categorical score in Langfuse. This is especially useful when the right answer is a label instead of a gradient: - Classify answers as `correct`, `partially_correct`, or `incorrect` - Mark support replies as `resolved`, `needs_follow_up`, or `escalate` - Label safety outcomes as `safe`, `needs_review`, or `blocked` Numeric scores are still the right fit for continuous dimensions like helpfulness, completeness, or faithfulness. But when your team needs explicit states for filtering, dashboards, or routing, categorical outputs are easier to interpret and act on. ## What's New - Choose `Numeric` or `Categorical` when creating a custom LLM-as-a-Judge evaluator - Define the allowed category values directly in the evaluator template - Optionally allow multiple matches when more than one label applies; Langfuse creates one score per selected category - View categorical results in evaluator logs and reuse them across Langfuse's existing score tooling ## Get started - [LLM-as-a-Judge Documentation](/docs/evaluation/evaluation-methods/llm-as-a-judge) - [What Are Scores?](/faq/all/what-are-scores) --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2026-03-20-categorical-llm-as-a-judge-scores.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .