Continuously Improving Agents With Langfuse

An agent can return 200s with normal latency and no exceptions while the user still does not get the help they needed. Because agents are nondeterministic, knowing that your software runs tells you little about whether its answers are any good.

In this workshop from AI Engineer, Lotte Verheyden and Annabell Schäfer build the online half of the improvement loop, tracing and monitoring, with a sample app called Specs, the Dad IT support agent.

Watch it if you have an agent in production, or close to it, and want to catch the conversations that go wrong. You can follow along in the free Langfuse workshop.

What you will learn

How the online part of the AI engineering loop (tracing and monitoring) connects to the offline part (datasets, experiments, evaluation)
What a trace needs to contain before it can support useful monitoring
How to pick monitoring signals that fit your own application
How to set up an LLM-as-a-judge evaluator and a code evaluator, and which part of the trace they should read
How to use the Langfuse CLI and Agent Skill so a coding agent can investigate your traces
Why datasets and evaluation criteria need regular maintenance

The AI engineering loop: online and offline

The workshop frames improvement as a loop with two parts:

Online: you trace your application in production and monitor those traces with dashboards, evaluators, and user feedback.
Offline: you build datasets of test cases, run experiments on proposed changes, and evaluate the results before you deploy.

Outside high-risk industries, the online part alone gets you far at the start. Find a problem during monitoring, make a simple fix, deploy, and watch the next traces come in. Once your product matures, or when mistakes have serious consequences, add the offline part. Your datasets come from what you saw in monitoring, and a change ships only once experiments show it is better without regressions.

This video covers the online part.

Build traces that are worth monitoring

If a trace is missing information, monitoring misses it too, your datasets get weaker, and you have less to go on when deciding what to evaluate. Get tracing right first.

A trace is made of observations, the steps your agent takes to produce an answer: LLM calls, tool calls, and other steps. Observations nest, so the tree mirrors how the agent ran.

What you record depends on the type of observation:

LLM calls: duration, token usage, cost, the model, the prompt version, and anything else that helps you judge that call.
Tool calls: what is relevant to the operation, such as the query for a web search.

Cost, latency, and tokens are captured per observation and roll up to the trace. See Get started with tracing to set this up, or ask your coding agent to do it with the Langfuse Agent Skill.

Three jobs of monitoring

Aggregate metrics. Cost, latency, and evaluator scores become trends over time, so you notice when quality drops. See metrics and dashboards.
Signal detection through user feedback. As traffic grows, you cannot read every trace. User feedback points you to the ones worth reading.
Feedback and evals. Automatic evaluators run on all your traces, but they mostly catch problems you already thought of. User feedback and human annotation scale less well, and they are where you find what you did not anticipate. Use both.

Meet Specs, the Dad IT support agent

Specs started as a side project for a dad who kept asking on WhatsApp how to set up his phone, his printer, and his computer. In the workshop it has a small web UI.

Specs is a TypeScript app built on the OpenAI Agents SDK. It uses the OpenAI API to generate replies and has two tools:

get_support_context to look up Dad's phone, operating system, and installed apps
search_help_library to search a set of frequently asked questions with guidance on common issues

To run it yourself, follow the setup lesson: clone the langfuse-workshop repository, copy .env.example to .env, create a Langfuse account (a project is created for you) and add your API keys, install dependencies, and start the app with npm run dev.

Ask a question like "How do I turn on my Bluetooth on my iPhone?" and a trace shows up in Langfuse. It is rooted in an agent observation, with generations and tool calls nested underneath. The sample app is already instrumented, so the workshop spends its time reading traces, not adding them.

Choose signals that fit your application

Start with what can go wrong in this app. You are not next to Dad when he uses Specs, so the traces have to tell you. For a support agent, the workshop starts with two signals:

User disagreement. Dad asks how to turn on Bluetooth, then replies "Where's the button?" That usually means the instructions were unclear or not detailed enough.
All-caps frustration. People who get frustrated with a chat app often start typing in all caps.

These are examples, not a list to copy. It often makes sense to check out the specific failure modes for your application. In chat-like applications, conversation signals are a good starting point for monitoring how people interact with your application.

To follow along, switch to the monitoring checkpoint with git checkout checkpoint/04-monitoring and open the monitoring lesson.

Evaluators for Specs

Langfuse has three kinds of evaluators:

Code evaluators handle deterministic checks. You write a Python or TypeScript function, and the same input always returns the same score.
LLM-as-a-judge evaluators handle what is implicit in a conversation. A model scores the trace and can explain the grade.
Decision model evaluators ask a typed question and write the answer as a score: a label, a level on a rubric, or a yes or no. Use them when the possible answers are known before you call the model.

The workshop covers the first two. To set one up, connect an evaluator model, choose the observations to target, and point the evaluator at the parts of the observation JSON it should see. The evaluator setup shows the steps in the UI.

The workshop sets up three evaluators for Specs:

User disagreement needs judgment, so it is an LLM judge on the agent observation. It reads the user's side of the conversation.
All-caps frustration is a plain text check, so it is a code evaluator. A short Python function returns true when the user message is written in all caps.
Out-of-scope requests need the system prompt as well as the latest user message, so this evaluator targets generation observations, where that prompt is available. Specs is scoped to phone support, and Dad also asks about a printer and a computer.

They run on live observations and attach their scores there, so you can filter for disagreement or all-caps frustration and review those conversations first. The same evaluators can run on experiments later.

Let a coding agent investigate your traces

Next, a script from the monitoring lesson seeds the project with example production traces and scores. That is more than you can read one by one.

Install the Langfuse CLI and the Langfuse Agent Skill, or ask your coding agent to install them. The CLI exposes the Langfuse API, so the agent can read your traces, scores, datasets, and experiment runs, and start new runs. The skill can also add tracing to an app.

Then ask your coding agent something like:

Please use the Langfuse skill to find issues in our traces.

The agent checks the scores you configured, such as disagreement and frustration, and also looks at latency, tool calls, and anything else that might be failing silently. On the seeded traces, its report found:

A silent retrieval failure in search_help_library. None of the evaluators checked for this, and nobody had noticed the tool was not returning useful information.
Retries adding latency and cost. Retrieval misses led to repeated tool calls.
An evaluation coverage gap. Only a subset of recent traces had evaluation scores at all.
A live run in Codex also flagged an issue with the prompt version.

Disagreement and all-caps scores tell you a user was unhappy, not whether retrieval, retries, or something else caused it. The tool observations answer that.

The coding agent also has your repository as context, so it can propose code changes, not only Langfuse configuration. In the workshop, the next request is to remove the search_help_library tool, since it does not seem to help. That change still needs testing before it ships, which is the offline part of the loop.

Close the loop

To keep improving an agent, maintain two things:

Your evaluation criteria. Check that evaluators are not over-indexing on one behavior and that they still tell you something useful.
Your datasets. Production shows you edge cases you did not plan for. Turn good production examples into dataset items and test changes against them. Experiments in CI/CD run that test on every change and catch regressions before you ship.

Keep doing this, so your dataset keeps matching what your users actually do.

If you start tomorrow

Use the Langfuse Agent Skill to add tracing to your repository.
Review traces manually to see what people actually do.
Build evaluators for your production traces around the failures that matter to you.
Use those signals to build datasets and experiments, and start optimizing offline.

Chapters

0:00 The AI engineering loop
3:08 Online and offline improvement
5:12 Build useful traces
6:38 Monitoring and user feedback
8:01 Meet Specs, the Dad IT support agent
10:06 Repository and account setup
12:28 Run the TypeScript application
13:23 Inspect a live support trace
15:02 Choose application specific signals
16:21 Configure the evaluator model
18:16 Set up a disagreement judge
19:16 Map trace messages into the evaluator
21:07 Code based all caps detection
24:16 Test and execute the monitors
24:58 Generate frustration and disagreement
26:13 Inspect evaluator scores
27:47 Detect out of scope requests
31:51 Seed production traces
33:50 Install the CLI and skill
35:24 Investigate traces with a coding agent
37:24 Retrieval, retries, and coverage findings
39:05 Propose a code change
40:03 Maintain datasets and close the loop
42:25 Workshop resources and next steps

Resources

Langfuse workshop: the full eight-module, self-paced version of this session, including prompt management, datasets, and experiments
Monitoring lesson and the langfuse-workshop repository
Langfuse Academy and the AI engineering loop
LLM-as-a-judge evaluators, code evaluators, and decision model evaluators
Conversation signals
Langfuse Agent Skill and Langfuse CLI
Datasets and experiments in CI/CD
Metrics and dashboards

Was this page helpful?

On this page