--- date: 2025-10-16 title: LLM-as-a-Judge Execution Tracing & Enhanced Observability description: Every LLM-as-a-Judge evaluator execution now creates a trace, allowing you to inspect the exact prompts, responses, and token usage for each evaluation. author: Hassieb ogImage: /images/changelog/2025-10-16-llm-as-a-judge-execution-tracing.png canonical: /docs/evaluation/evaluation-methods/llm-as-a-judge --- > **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) and the API/SDK reference (https://api.reference.langfuse.com). We're excited to announce a major enhancement to Langfuse's LLM-as-a-Judge evaluations: **full tracing of evaluator executions**. Every time an LLM-as-a-Judge evaluator runs, we now create a detailed trace that captures the complete LLM interaction, giving you unprecedented visibility into how your evaluations are performing. **What's New** Every LLM-as-a-Judge evaluator execution going forward is linked to a Langfuse trace of the underlying LLM call. This means you can: - **Debug evaluation prompts**: See exactly what prompt was sent to the judge LLM - **Inspect model responses**: View the complete response including score and reasoning - **Monitor token usage**: Track costs and performance for each evaluator execution - **Trace evaluation history**: Navigate from any score back to its source LLM interaction **How to access execution traces**: There are four ways to navigate to an evaluator execution trace: 1. **Score tooltip in trace view**: For LLM-as-a-Judge scores, hover over any score badge and click "View execution trace" ![Score tooltip with execution trace link](/images/docs/evaluation/eval-execution-traces-score-tooltip.png) 2. **Tracing table**: Filter the environment to `langfuse-llm-as-a-judge` to view all evaluator execution traces ![Tracing table filtered to langfuse-llm-as-a-judge environment](/images/docs/evaluation/eval-execution-traces-tracing-table.png) 3. **Scores table**: Enable the "Execution Trace" column in the scores table to see all evaluator executions ![Scores table with execution trace column](/images/docs/evaluation/eval-execution-traces-scores-table.png) 4. **Evaluator logs table**: View execution trace IDs in the evaluator logs for detailed execution history ![Evaluator logs with execution traces](/images/docs/evaluation/eval-execution-traces-logs-table.png) **Why This Matters** Previously, debugging failed evaluations or understanding why a judge gave a particular score required guesswork. Now, with full tracing: 1. **Trust your evaluations**: Verify that the judge received the correct input and made sound judgments 2. **Optimize costs**: Identify expensive evaluation patterns and optimize your prompts 3. **Faster debugging**: Instantly see what went wrong when an evaluation fails 4. **Audit trail**: Complete history of every evaluation decision for compliance and analysis **Getting Started** This feature is automatically enabled for all LLM-as-a-Judge executions going forward. ## Learn More - [LLM-as-a-Judge Documentation](/docs/evaluation/evaluation-methods/llm-as-a-judge) - [Understanding Evaluator Execution Traces](/docs/evaluation/evaluation-methods/llm-as-a-judge#trace-evaluator-executions) --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2025-10-16-llm-as-a-judge-execution-tracing.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .