--- title: Error analysis sidebarTitle: Error analysis description: A structured way to identify how your LLM application fails by reading traces and grouping the failures into categories you can act on. --- # Error analysis LLM app failures are usually domain-specific. A RAG system retrieves the wrong section of a document, a support bot misses a follow-up, an agent picks the wrong tool. The tone doesn't match what you expect in your specific context. While evaluator libraries can be a good starting point, you will find the deep insights into failure modes by reading actual traces of your system. ## What error analysis is Error analysis is a structured way to do that reading. The mechanic is borrowed from qualitative research: you read first, name what's broken in your own words, and let failure categories emerge from those notes rather than checking each trace against a predefined list. The output is a failure taxonomy that fits your application, paired with failure rates that tell you which categories matter most. The process is five steps: 1. **Gather traces.** Pull a representative sample from production traffic, a dataset, or experiment outputs. 2. **Open coding.** Read each trace and write a free-text note about the first thing that went wrong. No predefined categories yet — let the failures define themselves. 3. **Cluster.** Group similar observations into named failure categories. An LLM can draft a taxonomy from your notes; you refine the names and split anything that conflates two root causes. 4. **Label and measure.** Tag every trace in the sample against the taxonomy and compute failure rates per category — qualitative reading turns into a chart. 5. **Decide and act.** For each category, choose between a prompt or code fix, an evaluator that catches it on future traces, or monitoring for now. You walk away with a prioritized list of decisions tied to your actual data: what to change today, what to measure going forward, and what to keep watching. ## When to run it [#when-to-run-it] - **Before designing evaluators** - so your traces define what's worth measuring, not generic criteria like "helpfulness." - **After a prompt rewrite, model swap, or new feature** - failure distributions shift, and new categories show up. - **When [monitoring](/academy/monitoring) surfaces a pattern** - a drop in scores, recurring complaints, an unusual cluster of low-confidence responses. - **While iterating locally** - a small dataset of representative inputs is enough; you don't need production traffic to start. - **As a recurring practice** - the first taxonomy is never the final one. Re-run each round as your app evolves. ## What you get out of it **A taxonomy specific to your app.** Generic metrics rarely match what's actually failing. Categories you find by reading your own traces do. **A split between fix-once bugs and recurring patterns.** Some failures are obvious prompt issues you fix once and move on. Others need an evaluator to catch the next time they happen. Error analysis sorts each into the right bucket, so you don't build evaluators for problems a prompt change would have solved. **A measurable baseline.** Once traces are labeled, failure rates per category turn vague intuition ("the bot seems worse since the last prompt update") into something you can chart and watch shift as you ship changes. ## How to run error analysis for your application > **Guide: [Error analysis](/guides/cookbook/error-analysis-llm-applications)** > > Select your sample data, build an annotation queue, cluster failure categories, quantify failure rates, and decide what to do. **Run with your agent** Paste this prompt into your coding agent. The Langfuse skill runs every step alongside you - pulling traces, clustering, computing failure rates. You make the domain calls. ```text I want to do a systematic error analysis of my LLM application to understand how it fails. Please install the Langfuse skill (https://github.com/langfuse/skills/tree/main/skills/langfuse) and the Langfuse CLI (https://github.com/langfuse/langfuse-cli), then guide me step by step through error analysis. ``` ## What comes next Categories you can fix directly become prompt updates or bug fixes. For the rest, [choosing what to evaluate](/academy/evaluate/choosing-what-to-evaluate) covers which categories deserve a standing metric and how to structure the set. [Datasets](/academy/datasets) hold the inputs you test against, and [evaluation](/academy/evaluate) is where you pick the method - code evaluators, LLM-as-a-judge, or human review - for each category. --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/academy/monitoring/error-analysis.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .