Jev-based evals are now available in Langfuse. Jev-like models give you and your team a way to run evals on production traffic at scale and identify signals from agent executions and user interactions. All of this at up to 40 to 400x lower cost than with LLM-based approaches.

Jev is part of a new model category that prioritizes performance on one dimension over general ability of the model. It can be seen as a "zero-shot classifier". As founder Diogo Almeida said on the latest Latent Space episode, they optimize for "Intelligence per Dollar" and design the model with fast, structured, machine-native decision-making inside software in mind.

While the category is being formed, we are excited to bring Jev-based evals into Langfuse already today.

What is Jev

Jev, TypeSafe's new model, handles narrow, typed questions over unstructured data. It is best suited for quick decisions that are repeated, high volume, and the possible answers are known before the call.

Jev offers three different output types:

• Choice picks one option from a set you define, up to 255, and returns the probability of each plus a confidence value.
• Score rates the state against ordered rubric levels and returns a probability-weighted value, the full distribution, and a confidence value. Up to 10 levels.
• Noul answers yes or no and returns the probability it is true. It carries no separate confidence field, so code that reads answer.confidence on everything will break on binaries.

Why is Jev exciting for evals

Jev's key strengths align well with best practices for evaluating your AI application, while overcoming some key issues of LLM-as-a-judge setups.

Significantly cheaper than using LLM as a judge at scale

Jev allows you to run evals on production traces at scale. The input tokens are up to 400x cheaper than frontier models. Where sampling was previously the strategy for cost efficient production monitoring, the full traffic can now be evaluated.

Allows for concurrent questions

The same state can be used across multiple questions. All questions are handled concurrently and independently. Where previously context creep and interaction effects of multiple LLM-based judgments interfered, Jev now treats it as separate assessments.

Forces you to think in distinct categories

The predefined output structures force you to think in distinct decisions during setup. You define a clear atomic question and in the output options need to specify the conditions for each verdict. As TypeSafe says, Jev is judging what you say, not what you mean.

While execution speed is not a bottleneck in async production evals, this new model category also allows for significantly faster execution.

Setting up Jev for production scoring in Langfuse

Setting up Jev-based evals is possible via the Evaluators tab in the Langfuse app.

You can also check out our setup in the demo project.

---

---