---
title: "Beginner's Guide to RAG Evaluation with Langfuse and Ragas"
description: "Webinar by Prof. Tom Yeh on how to evaluate RAG applications with Langfuse and Ragas."
ogImage: /images/videos/beginners-guide-to-rag-evaluation.jpg
---

# Beginner's Guide to RAG Evaluation with Langfuse and Ragas

We recommend this webinar by Prof. Tom Yeh on evaluating Retrieval
Augmented Generation (RAG) applications. It provides an excellent introduction
to RAG and explains how Langfuse can help debug and evaluate RAG systems,
particularly when combined with [Ragas](https://docs.ragas.io) metrics.

- Presenter: Tom Yeh, Associate Professor at University of Colorado Boulder
- Resources: [Webinar slides](https://aibyhand.substack.com/p/beginners-guide-to-rag-evaluation) on Tom's blog

<iframe
  width="100%"
  className="aspect-video rounded mt-10"
  src="https://www.youtube-nocookie.com/embed/2fqs8Nlh5UI?si=A6CYkwA0iGs-7uak&amp;start=537"
  title="YouTube video player"
  frameborder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
  referrerpolicy="strict-origin-when-cross-origin"
  allowFullScreen
></iframe>

## Our Notes

### 1. RAG Overview

- User inputs a question.
- Instead of directly querying a large language model (LLM), the input is augmented with context retrieved from a database.
- The augmented query is then sent to the generator to produce a response.

### 2. Components of RAG

- Retriever: Fetches relevant context from a database.
- Augmentation: Combines user query with retrieved context.
- Generator: Produces an answer based on the augmented query.

### 3. Evaluation of RAG Systems

- Trace Analysis:
  - Tracing the steps from user input to final output to understand system performance.
  - Involves logging each step such as retrieval, augmentation, and generation.
- Metrics:
  - Conciseness: Measures how succinct an answer is.
  - Helpfulness: Evaluates the usefulness of an answer.
- Tools:
  - [Langfuse](/): Used for tracing and logging RAG operations.
  - [RAGAS](https://docs.ragas.io) (RAG Assessment): Provides detailed metrics for evaluating RAG systems.

### 4. Langfuse

- [Features](/docs):
  - Logs each step in the RAG process.
  - Provides a timeline view of operations.
  - Allows for comparison of different interactions.
  - Many additional LLM Ops features such as prompt management, cost analysis, benchmarking, and more.
- Demo:
  - Demonstrated a chatbot application using Langfuse for tracing and logging interactions. Public link: [langfuse.com/demo](https://langfuse.com/docs/demo)
  - Showed how to analyze the performance of the RAG system using Langfuse metrics.

### 5. RAGAS

- [Metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.html):
  - Faithfulness: Accuracy of the generated answer based on retrieved context.
  - Hallucination: Incorrect information generated by the model.
  - Answer Relevancy: Relevance of the generated answer to the original question.
  - Context Recall: Ability to retrieve all relevant information.
  - Context Precision: Accuracy of the retrieved context.
- Implementation:
  - Uses prompts to evaluate faithfulness and relevancy.
  - Ground truth data is used to evaluate retrieval metrics.

### 6. Cost Considerations

- Evaluating the cost associated with using large language models for generation and evaluation.
- Emphasized the need to balance between expensive and cheaper models for different tasks.

### 7. Additional Metrics

- Mentioned other metrics like context utilization, context entity recall, and noise sensitivity.
- Highlighted the importance of choosing the right metrics based on specific needs and explaining them to stakeholders.

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/guides/videos/beginners-guide-to-rag-evaluation.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.