Decision model evaluators
Decision models answer typed questions about an observation instead of generating free-form text. Each Choice, Score, or Yes / no question writes one score.
They are often cheaper to run than LLM-as-a-Judge, which is why you would use them when you do not need written reasoning. Use an LLM judge when you do, or code evaluators for deterministic checks.
![]()
Set up step-by-step
Create a decision-model evaluator
Go to the Evaluators page and click New evaluator. Choose New decision model evaluator, then select a connection and model. Add the connection under Settings → LLM Connections if you do not have one yet.
![]()
Define the questions
Add one question per criterion. For each question, pick the type, write the instruction, define the allowed answers, and set the score name. One evaluator can ask several questions in a single call.
| Question type | Langfuse score | Example |
|---|---|---|
| Choice | Categorical score with the selected option | "Which team should handle this ticket?" → billing, technical, sales |
| Score | Numeric score with the expected level, 0 to levels - 1 | "How frustrated is the customer?" → 0 calm … 3 angry |
| Yes / no | Numeric score with P(true) between 0 and 1 | "Does the message request a refund?" → 0.93 |
Keep each question atomic: one decision per question. If you are asking "is the answer correct and polite?", split it into two. Both run in the same call.
Map the model input
Map the observation data the model should evaluate. How you map the input depends on the provider; see provider-specific guidance.
Test the evaluator
On the right, filter to representative sample observations, select one, and run the evaluator. The test panel shows one row per question with its probability distribution and confidence where available, along with the estimated cost. Use Raw output to inspect the provider request.
Save the evaluator
After saving, you can:
- Create a rule from the filters you used to select test samples, or attach the evaluator to an existing rule to run it on incoming observations.
- Continue without a rule. You can still use the evaluator for batch evaluation or prompt experiments.
Connect a provider
Add an LLM connection under Settings → LLM Connections if you do not have one yet. Select that connection and a decision model when you create the evaluator.
Provider-specific guidance
FAQ
Why is my Yes / no score a number and not a boolean?
Decision models return the probability that the statement is true, not a hard label. Langfuse stores that probability as a numeric score between 0 and 1 so you keep the calibration. Choose a threshold when you filter or alert, for example >= 0.7. If you need a boolean, use a Choice question with yes and no.
Can I use decision-model evaluators in experiments and batch evaluation?
Yes. They run on incoming observations through rules, on historical observations through batch evaluation, and on experiments. Map the experiment fields your questions need into the provider input.
How do I debug a decision-model run?
Every run creates an execution trace in the internal environment langfuse-llm-as-a-judge, named Execute evaluator: <evaluator name>. Internal environments are hidden from the default tracing view: filter by environment = langfuse-llm-as-a-judge or open the trace from a score.
Probabilities and confidence are stored in score metadata where the provider returns them, not in the score value.
What are the limits?
| Constraint | Limit / guidance |
|---|---|
| Questions per evaluator | 1 to 50, all answered in one call. |
| Choice options | 2 to 255 options with unique values. |
| Score levels | 2 to 10 ordered levels. |
Input and questions are sent through the selected connection. Review your provider terms, and use masking where needed. Input size limits and refusal behavior depend on the provider; see provider-specific guidance. Keep the input focused on the data needed for the decision.
Related resources
- Catching conversation signals in Langfuse: score production chat with the User Conversation Signal template.
- Using TypeSafe's Jev for evals: what Jev is good for and a walkthrough of Jev-as-a-judge.
- Writing good evaluators: how to define criteria that hold up.
- TypeSafe documentation: question primitives, model versions, and known limitations.
GitHub Discussions
Last updated on