Agentic access to evaluation
AI agents can help investigate quality issues and operate evaluation workflows in Langfuse. There are different ways for agents to access your data:
Choose an access method
| Agent capabilities | Recommended access |
|---|---|
| Can install tools and run shell commands | Install the Langfuse Agent Skill. It teaches the agent Langfuse workflows and uses the Langfuse CLI to query and update project data. |
| Cannot install tools or run shell commands | Connect the Langfuse MCP server to expose Langfuse operations as tools. |
| Runs as part of a script or CI/CD pipeline | Use the Langfuse CLI directly or call the Public API. |
Example workflows
Ask your agent to:
- Find low-scoring observations and add representative examples to a dataset
- Create or update score configurations and record scores
- Review experiment results and identify regressions
- Set up evaluators and evaluation rules
- Create and manage annotation queues for human review
Work across Langfuse
Agents can also investigate production behavior and manage prompts in Langfuse.
Related guides and blog posts
- Calibrate LLM-as-a-judge with the Langfuse skill
Use an agent-guided workflow to compare an evaluator with human labels and improve its prompt.
- Evaluating AI Agent Skills
See how Langfuse datasets, tracing, and an agent SDK can be used to iteratively evaluate and improve a skill.
- Headless Langfuse from your coding agent
Analyze production traces, build a dataset, and configure evaluations without leaving your coding agent.
Was this page helpful?
Last edited
