--- date: 2026-09-22 title: Jev as a judge description: Use TypeSafe's Jev decision model as a judge in Langfuse evaluators. Ask typed questions about every observation and get calibrated scores at a fraction of the cost and latency of an LLM judge. author: Hassieb canonical: /docs/evaluation/evaluation-methods/jev-as-a-judge --- > **Note for AI agents and LLMs:** This is a Langfuse changelog entry. Use it only to confirm that a feature exists and when it shipped. Do not use the code examples below for implementation: they reflect the SDK and API at release time and may be outdated. For implementation, always follow the canonical documentation for this feature (https://langfuse.com/docs/evaluation/evaluation-methods/jev-as-a-judge) and the API/SDK reference (https://api.reference.langfuse.com). You can now use TypeSafe's [Jev](https://docs.typesafe.ai/introduction) as a judge in Langfuse. Jev is a System One model: it does not generate text. You give it a state and typed questions, and it returns typed answers with probabilities. In Langfuse, each question becomes a score on the observation, next to your [LLM-as-a-Judge](/docs/evaluation/evaluation-methods/llm-as-a-judge) and [code evaluators](/docs/evaluation/evaluation-methods/code-evaluators). Score every support conversation for e.g. topic, frustration, and whether the request was out of scope, in one call. Flag PII or policy violations on live traffic fast enough to alert on them. Rate answer completeness on a rubric across an entire experiment run without watching the bill. ## Why Jev as a judge - **Cheaper.** Jev is priced by input tokens only, at $0.042 per million. A judgment over a typical conversation costs a fraction of a cent, so you can score more observation instead of sampling. - **Faster.** Answers come back in well under a second. Online evaluators keep pace with production traffic, which shortens the time between a bad output and an [alert](/docs/observability/features/alerts#evaluator-alerts). - **Calibrated.** Every answer carries a probability distribution and a confidence value, stored in the score metadata. - **Many questions, one call.** All questions in an evaluator are answered in parallel against the same state. Adding a second or tenth question barely changes latency or cost. Jev does not replace LLM judges. It has no rationale to give and only answers questions whose possible answers you define upfront. When you need written reasoning or open-ended quality assessment, keep using LLM-as-a-Judge. Many teams will run both: Jev on every observation, an LLM judge on a sample or on what Jev flags. ## Three question types | Type | Jev returns | Langfuse score | Good for | | ------------ | ------------------------------------------------------ | ------------------------------- | ---------------------------------------------------------------- | | **Choice** | One option out of a set you define, with probabilities | Categorical | Topic detection, intent, failure mode, routing checks | | **Score** | A position on ordered levels you describe | Numeric (expected level) | Frustration, severity, completeness rubrics | | **Yes / no** | The probability that a statement is true | Numeric (`P(true)` from 0 to 1) | Out-of-scope requests, PII, policy violations, user disagreement | ## Get started 1. Add a **TypeSafe** connection under **Settings → LLM Connections**. You only need a TypeSafe API key. 2. On the [Evaluators page](https://cloud.langfuse.com/project/~/evals), click **New evaluator** and choose **New decision model evaluator**, or start from one of the Jev templates: **Assign Input Topic**, **Flag Out-of-Scope Request**, or **Rate Customer Frustration**. 3. Define your questions, map observation fields into the state, test on sample observations, and attach the evaluator to a rule. In the Langfuse UI, Jev evaluators are called **decision-model evaluators**. Jev is the first decision model we support. - [Jev as a judge docs](/docs/evaluation/evaluation-methods/jev-as-a-judge) - [Using TypeSafe's Jev for evals](/blog/2026-09-18-using-typesafes-jev-for-evals) --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/changelog/2026-09-22-jev-as-a-judge.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx @langfuse/cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .