Monitoring
How monitoring fits into the loop
Tracing provides a complete record of what your LLM app is doing — every request, every model call, every tool use. Monitoring is how you make sense of that data: you decide what to watch, get that data onto your traces, and use it in two ways, to track how your system performs over time and to find the specific traces worth investigating. Together, they shift you from having data to actually understanding your system well enough to improve it.
What should you monitor?
Operational metrics
Cost, latency, token usage, and error rates tell you whether your application is healthy and affordable to run. They are the first to show when a model update, a prompt change, or a traffic spike makes your application slower, more expensive, or less reliable, and they are what you set budgets and service levels against. Most of them are standard fields on your traces, so you get them as soon as you instrument your application.
Quality scores
Quality scores measure the quality of your agent's execution according to criteria you define, such as whether a response is grounded in the retrieved documents or follows your support policy. They let you keep watching failure modes you want to track, and catch quality drifting as models and usage change. Choosing what to evaluate covers which criteria deserve a standing metric.
User feedback signals
User signals are things users do or say that show you how they perceived an agent's output: a thumbs down, a retry, a request for a human, a draft sent without edits. They measure quality as perceived by the user rather than evaluating actual quality. They are useful because they are often cheap and straightforward to implement, and they surface failures you didn't think to evaluate for yourself. Capturing user signals covers the kinds of signals, how to read them, and their biases.
Multiple ways of getting this data onto your traces
Standard fields come with instrumentation: latency, token usage and cost, model, and errors, plus the metadata, tags, users, and sessions you set when tracing. Most operational metrics need nothing more.
Scores hold everything else. You attach them in three ways:
- Manually, by reviewing traces and scoring them in the UI or working through an annotation queue, for example during error analysis. It's how you learn what good looks like, and it gives you labels to check automated evaluators against.
- With evaluators that score production data automatically. There are different kinds of evaluators, and arriving at a good set of metrics to measure the quality of your agent is extremely important. More on that in the Evaluate section.
- Via the API or SDK when your application captures something, such as a thumbs-down click, an edited draft, or a support ticket that reopens. This is how most user signals are recorded.
Different monitoring practices
There are different kinds of monitoring, and in a healthy setup, you should be doing a mix of these.
Track aggregate metrics
Aggregates tell you whether things are getting better or worse. Chart cost, latency, quality scores, and signal rates on dashboards to answer questions like: Did your latest prompt change improve anything? Is quality drifting as usage grows? Set alerts on the metrics where a spike needs attention right away, so you don't have to watch the dashboards to catch it.
Investigate traces with error analysis
Looking at your traces yourself is the most important thing you can do to understand what is going on in your application. Every decision you make, especially in the beginning, is downstream of this: which quality scores to set up, which user signals to capture, and what to fix first. Errors, low quality scores, and negative user signals help you find interesting traces to look at, and you can do error analysis on these to decide what you want to automatically evaluate.
Where to start
Start small and build your monitoring setup from real traces rather than abstract ideas about what might matter.
-
Start by looking at your data manually. Read through traces and notice what kinds of things keep appearing. You can't set up useful monitoring before you know what you're looking for.
-
Use error analysis to surface what's worth tracking. Error analysis gives you a structured way to find patterns across your traces, the kinds of recurring issues worth turning into automated evaluators you can run continuously.
-
Capture the user signals your application already produces. Think about how your users show that something went wrong: a user disagreeing in a support chat, a correction in a process automation flow. These user signals are often more actionable than generic scores, are cheap to capture, and point you to traces worth reading without manual labeling.
-
Treat it as an iterative process. A working monitoring setup isn't something you configure once and leave. Usage patterns shift, models get updated, new failure modes emerge. Keep refining your setup so you can cut through the noise and stay focused on what actually matters.
What comes next
When monitoring surfaces something worth investigating, you have a few options: fix it directly if the cause is obvious, capture it in a dataset if it looks like a pattern, or run a structured evaluation if you suspect something systemic. Which path you take depends on how confident you are about the cause.
- Datasets: capturing production traces for evaluation
- Experiments: testing whether a fix actually worked
Last updated on