Langfuse launches AI Gateway Learn more
DocsDecision model evaluators

Decision model evaluators

Decision models answer typed questions about an observation instead of generating free-form text. Each Choice, Score, or Yes / no question writes one score.

They are often cheaper to run than LLM-as-a-Judge, which is why you would use them when you do not need written reasoning. Use an LLM judge when you do, or code evaluators for deterministic checks.

Decision-model evaluator with a Score question, mapped input, and live test results

Set up step-by-step

Create a decision-model evaluator

Go to the Evaluators page and click New evaluator. Choose New decision model evaluator, then select a connection and model. Add the connection under Settings → LLM Connections if you do not have one yet.

New decision model evaluator in the template gallery, with a Yes / no question

Define the questions

Add one question per criterion. For each question, pick the type, write the instruction, define the allowed answers, and set the score name. One evaluator can ask several questions in a single call.

Question typeLangfuse scoreExample
ChoiceCategorical score with the selected option"Which team should handle this ticket?" → billing, technical, sales
ScoreNumeric score with the expected level, 0 to levels - 1"How frustrated is the customer?" → 0 calm … 3 angry
Yes / noNumeric score with P(true) between 0 and 1"Does the message request a refund?" → 0.93

Keep each question atomic: one decision per question. If you are asking "is the answer correct and polite?", split it into two. Both run in the same call.

Map the model input

Map the observation data the model should evaluate. How you map the input depends on the provider; see provider-specific guidance.

Test the evaluator

On the right, filter to representative sample observations, select one, and run the evaluator. The test panel shows one row per question with its probability distribution and confidence where available, along with the estimated cost. Use Raw output to inspect the provider request.

Save the evaluator

After saving, you can:

  • Create a rule from the filters you used to select test samples, or attach the evaluator to an existing rule to run it on incoming observations.
  • Continue without a rule. You can still use the evaluator for batch evaluation or prompt experiments.

Connect a provider

Add an LLM connection under Settings → LLM Connections if you do not have one yet. Select that connection and a decision model when you create the evaluator.

Provider-specific guidance

FAQ

Why is my Yes / no score a number and not a boolean?

Decision models return the probability that the statement is true, not a hard label. Langfuse stores that probability as a numeric score between 0 and 1 so you keep the calibration. Choose a threshold when you filter or alert, for example >= 0.7. If you need a boolean, use a Choice question with yes and no.

Can I use decision-model evaluators in experiments and batch evaluation?

Yes. They run on incoming observations through rules, on historical observations through batch evaluation, and on experiments. Map the experiment fields your questions need into the provider input.

How do I debug a decision-model run?

Every run creates an execution trace in the internal environment langfuse-llm-as-a-judge, named Execute evaluator: <evaluator name>. Internal environments are hidden from the default tracing view: filter by environment = langfuse-llm-as-a-judge or open the trace from a score.

Probabilities and confidence are stored in score metadata where the provider returns them, not in the score value.

What are the limits?
ConstraintLimit / guidance
Questions per evaluator1 to 50, all answered in one call.
Choice options2 to 255 options with unique values.
Score levels2 to 10 ordered levels.

Input and questions are sent through the selected connection. Review your provider terms, and use masking where needed. Input size limits and refusal behavior depend on the provider; see provider-specific guidance. Keep the input focused on the data needed for the decision.

GitHub Discussions


Was this page helpful?

Last updated on