Evaluate Production Traffic
This guide walks you through scoring live production traces in Langfuse. If you don't yet know what to evaluate, start with Choosing what to evaluate. For how evaluators, scores, and rules fit together, see Core Concepts.
Agentic installation
Install the Langfuse Agent Skill to let your coding agent access all Langfuse features.
Ask your coding agent to install the skill by pointing to the GitHub repository and instruct it to get started with online evaluation.
Install the Langfuse Agent Skill from github.com/langfuse/skills
and use it to set up online evaluation for this application
with Langfuse.Langfuse has a Cursor Plugin that includes the skill automatically.
Then prompt your agent:
Set up online evaluation for this application with Langfuse.Install via npm (skills CLI):
npx skills add langfuse/skills --skill "langfuse"If you want to target a specific agent directly:
npx skills add langfuse/skills --skill "langfuse" --agent "<agent-id>"Alternatively you can manually clone the skill
- Clone repo somewhere stable
git clone https://github.com/langfuse/skills.git /path/to/langfuse-skills- Make sure your agent's skills dir exists
mkdir -p /path/to/<agent-skill-root>/skills- Symlink the skill folder
ln -s /path/to/langfuse-skills/skills/langfuse /path/to/<agent-skill-root>/skills/langfuseThen prompt your agent:
Set up online evaluation for this application with Langfuse.Manual setup
This path sets up online evaluation in the Langfuse UI. You create an evaluator (how to score), test it on real observations, then attach a rule (which incoming observations to score).
Create an evaluator
Open the Evaluators page and click New evaluator. Choose a template or create an evaluator from scratch.
- LLM-as-a-Judge uses an LLM to score. Use this for qualities that need language understanding, such as relevance, tone, or whether a request is out of scope. You need an LLM connection first.
- Code evaluator is a Python or TypeScript
evaluatefunction. Use this for deterministic checks such as JSON validity, required fields, or keyword rules.
Test on sample observations
On the right, filter to representative production observations, select one, and run the evaluator to test and iterate until the result matches what you would expect.
Attach a rule to incoming traces
After you save the evaluator, create a rule from the filters you used while testing, or attach the evaluator to an existing rule.
A rule defines which incoming observations are scored: filters, sampling rate, and one or more evaluators. Review the matching volume from the past seven days. For LLM-as-a-Judge, also review the estimated cost and lower the sampling rate if needed.
See scores on production traces
New matching observations receive scores as they arrive. Open a scored observation to inspect the value and the judge's reasoning, then use score analytics or a custom dashboard to watch the metric over time.
You can also run the same evaluator on selected historical observations with batch evaluation.
Other ways to score live traffic
Automated evaluators are the fastest way to score production continuously. These other methods also attach scores to live data:
| If you want to... | Use |
|---|---|
| Review a sample of traces manually | Scores via UI, annotation queues |
| Capture thumbs-up/down or other feedback signals from users | User feedback |
| Push scores from your own pipeline or agent | Scores via API/SDK |
Next steps
- Read the Langfuse Academy evaluation module to decide what to measure and how to write evaluators you can trust
- Set an alert when a score drops below a threshold
- Analyze quality in custom dashboards
Last edited
