---
title: "LLM Analytics 101 — How to Measure and Improve Your LLM Application"
seoTitle: "LLM Analytics 101: Measure and Improve"
description: "A comprehensive guide to LLM analytics. Learn how to trace, measure, and analyze key metrics — volume, cost, latency, quality, and errors — to improve your LLM application."
tags: [article]
---

# LLM Analytics 101 — How to Measure and Improve Your LLM Application

  ![LLM Analytics 101](/images/blog/faq/llm-analytics.png)

_This guide gives builders on the LLM application layer an understanding of the **why**, **what** and **how** of tracing & analytics to improve their LLM applications_

## LLMs Have Changed Software Delivery

Generative AI outputs are not deterministic. That is, they cannot be reliably forecasted. This changes how software is delivered as compared to more 'traditional' software engineering. If it is not clear what an output will look like and what a '_good_' output is, it is harder to assure quality and build robust tests before shipping code.

Learning from production data has taken the place of extensive software design and testing on the LLM application layer. But to learn from production, you have to trace your LLMs and analyze what works and what does not.

## Tracing LLM apps - What's Different?

Building LLM-based apps means integrating multiple complex elements and interactions to your code. This can mean chains, agents, different base models, tools, embedding retrieval and routing. Traditional logging and analytics tools are not well equipped to ingest, display and analyze these new ways of interacting with LLMs.
The new logging stack needs to think LLM-native from the ground up. That means grouping calls and visualizing them in a way that enables teams to understand and debug them.

## Let's Dive in: What to Measure?

```ts
// Example generation creation
import { startObservation } from "@langfuse/tracing";

const generation = startObservation(
  "chat-completion",
  {
    model: "gpt-3.5-turbo",
    modelParameters: {
      temperature: 0.9,
      max_tokens: 2000,
    },
    input: messages,
  },
  { asType: "generation" },
);
```

The baseline requirement to improve an LLM-based app is to trace its activity. But what does that mean and what do I want to record? From working with our users at the bleeding edge of LLMs, we've see five metrics emerge to keep track of:

- **Volume:** The foundation for all other metrics - track all LLM calls and their content and attach relevant metadata for both prompts and completions.
- **Costs:** Record token counts and pricing to compute the cost of each call. Track GPU seconds and pricing for self-hosted models.
- **Latency:** Measure latency for every call. Use this data to analyze which steps add latency and start improving your users' experience.
- **Quality:** Proactively solicit user feedback, conduct manual evaluations and score outputs using model-based evaluations.
- **Errors/Exceptions:** Monitor for timeouts and HTTP errors, such as rate limits, that are indicative of systemic issues.

## Implementing Effective Analytics through KPIs

We've seen successful teams implement the following best practice KPIs by slicing the above five metrics (volume, cost, latency, quality, errors) by:

- **Use case:** Cluster prompts and completions by use case to understand how your users are interacting with your LLM
- **Model and configuration:** How do different models and model configurations affect quality, latency or errors?
- **Chain and step:** Drill down into chains to understand what drives performance
- **User data:** Group users by specific characteristics to gain insight into personas and specific constituencies in your product
- **Chain and step:** Drill down into chains to understand what drives performance
- **Model and configuration:** Track how different models and model configurations affect quality, latency or errors
- **Use case:** Cluster prompts and completions by use case to understand how your users are interacting with your LLM
- **Time:** Inspect your KPIs over time and detect trends
- **Version:** Track prompts, chains and software releases by their version and understand performance changes
- **Geography:** Especially important for latency
- **Language:** Understand how well your app works by user language

## Step-by-Step: Implementing Tracing & Analytics in LLM Applications

1. **Define goals:** What do you want to achieve and how do your goals align with your users' requirements. Take the above metrics as a starting point to define KPIs unique to your application.
2. **Incorporate tracking:** This means backend execution and scores (e.g. capturing user feedback in the frontend).
3. **Inspect and debug:** Understand your users by inspecting runtime traces through a visual UI
4. **Analyze:** Start by measuring cost by model/user and time, cost by product feature, latency by step of a chain and start scattering quality/latency/cost grouped by experiments or production versions.

## Give Langfuse a Spin

[Langfuse](/) makes tracing and analyzing LLM applications accessible. It is an open-source project under MIT license.

It offers data integration with async SDKs (JS/TS, Python), via API, and Langchain integrations. It provides a UI for debugging complex traces & includes pre-built dashboards to analyze quality, latency and cost. It allows for recording user feedback and using LLM models to grade and score your outputs. To get going, refer to the [quickstart guide](/docs/observability/get-started) in the docs.

Visit us on [Discord](/discord) and [GitHub](https://github.com/langfuse/langfuse/) to engage with our project.

![A trace in Langfuse](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b3ksrdnl61f7vzaej4t6.png)
Interested? Sign up to try the demo at [langfuse.com](/). Self-hosting instructions can be found in [our docs](/self-hosting).

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/faq/all/llm-analytics-101.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.