How to evaluate RAG faithfulness with LLM-as-a-judge

TL;DR: Faithfulness measures whether a RAG answer is grounded in the retrieved context: extract the factual claims the answer makes, check each claim against the retrieved documents, and score the share of supported claims. It is reference-free, so it needs no ground-truth answers, which makes it the RAG metric you can run on any dataset and on live traffic. This page defines the metric, shows how to design a faithfulness judge prompt, and implements it end to end with the Langfuse Python SDK: as an offline experiment with run_experiment, and as a managed LLM-as-a-judge evaluator on production traces.

What RAG faithfulness measures

Faithfulness (also called groundedness) is the degree to which an answer is supported by the context that was retrieved for it. A faithful answer only states things the retrieved documents say or directly imply. An unfaithful answer mixes in claims from the model's parametric knowledge, or invents details outright, even when the prose reads well and the answer happens to be correct.

The standard formulation, popularized by Ragas, is claim-based:

faithfulness = number of claims supported by the retrieved context / total claims in the answer

A score of 1.0 means every claim traces back to the context. A score of 0.5 means half the answer is unsupported. Because the comparison target is the retrieved context, not a ground-truth answer, faithfulness is a reference-free metric: you can compute it for any question, including ones nobody has written an expected answer for.

Faithfulness is one of three metrics that get conflated in RAG evaluation, and they answer different questions:

An answer can be perfectly faithful and useless: "our refund policy covers orders" is grounded in a refund document but does not answer "how long do I have to return this?". That failure belongs to answer relevance evaluation. Faithfulness is also narrower than hallucination detection: it is the RAG-specific, context-bounded version of the problem, checked against documents you control rather than against world knowledge.

How faithfulness is measured

Faithfulness requires reading comprehension, so it is measured with an LLM judge rather than string matching. Deterministic checks like citation presence or lexical overlap are useful pre-filters, but they cannot tell a paraphrase from a fabrication. The measurement pipeline, independent of tooling, has three steps:

1. Decompose the answer into claims. Each atomic factual statement in the answer becomes one unit of evaluation. "Refunds take 5 business days and go to the original payment method" is two claims.
2. Verify each claim against the retrieved context. The judge model receives the context and one or more claims, and decides per claim whether the context states or directly implies it. World knowledge does not count as support.
3. Aggregate. The share of supported claims is the faithfulness score. The list of unsupported claims is the debugging payload: it tells you which sentence to fix and whether the failure is in generation (context contained the fact, model ignored it) or retrieval (context never contained it).

To run this you need the three inputs on hand at evaluation time: the question, the generated answer, and the exact context that was retrieved for that answer. This is why faithfulness evaluation and tracing are naturally coupled. If your LLM-as-a-judge setup can see the retrieved documents next to the generation, faithfulness is a prompt away; if the context was thrown away after generation, there is nothing to verify against.

Designing a faithfulness judge prompt

The judge prompt determines whether your faithfulness scores are trustworthy. Two design decisions matter most.

Claim extraction beats holistic judgment. A holistic judge ("rate the faithfulness of this answer from 1 to 5") produces scores that drift between runs and hide what failed. A claim-extraction judge makes many small binary decisions instead: list the claims, label each one supported or unsupported. Each decision is easier for the judge model than a global rating, the aggregate score has a defined meaning (share of supported claims), and the unsupported-claim list localizes the failure for whoever reads the score comment.

Binary decisions beat graded scales. Langfuse Academy's evaluation module recommends binary pass/fail scores over 1-to-5 scales for evaluator design in general, because binary forces a definition of acceptable versus unacceptable while graded scales leave "what separates a 3 from a 4" undefined. Claim-level faithfulness gets you both properties: every judge decision is binary, and the numeric score comes from aggregation rather than from a rating the judge produces directly.

Beyond those two decisions, the prompt rules that prevent the common failure modes:

• Define what counts as a claim. Restrict extraction to factual statements. Hedges, questions back to the user, and statements of inability ("I could not find this in the documentation") are not claims and must not be penalized, otherwise the judge punishes exactly the honest behavior you want from a RAG system.
• Forbid outside knowledge explicitly. Judges default to marking true statements as supported. State that a claim counts as supported only if the context says or directly implies it, even if the claim is true.
• Allow paraphrase. Support does not require verbatim overlap. Without this rule the judge under-scores answers that correctly summarize the context.
• Demand structured output. Have the judge return JSON with one entry per claim, and compute the score in code. Never ask the judge to do the arithmetic.
• Pin the judge's temperature to 0 so repeated evaluations of the same answer agree.

Run a faithfulness experiment with the Langfuse Python SDK

The script below is self-contained: it creates a dataset of questions, runs a small RAG task against it, and scores every answer with a claim-extraction faithfulness judge using the Python SDK v4 experiment runner. The task returns both the answer and the contexts it used, so the evaluator receives everything it needs through the standard experiments via SDK contract. Replace the toy corpus and retrieval function with your vector store.

# pip install langfuse openai
import json

from langfuse import Evaluation, get_client
from langfuse.openai import OpenAI  # drop-in OpenAI client, traced automatically

# Reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST from the environment
langfuse = get_client()
client = OpenAI()

# --- A minimal corpus and retriever standing in for your vector store ---

DOCUMENTS = [
    "Orders can be refunded within 30 days of delivery. Refunds are issued to "
    "the original payment method within 5 business days.",
    "Standard shipping takes 3 to 5 business days within the EU. Express "
    "shipping delivers within 24 hours for orders placed before 2pm.",
    "All devices include a 2-year manufacturer warranty covering hardware "
    "defects. Water damage is not covered by the warranty.",
]

def retrieve(question: str) -> list[str]:
    words = question.lower().split()
    ranked = sorted(DOCUMENTS, key=lambda d: -sum(w in d.lower() for w in words))
    return ranked[:2]

# --- One-time setup: create the dataset in Langfuse ---

langfuse.create_dataset(name="rag-faithfulness-demo")
for question in [
    "How long do I have to return an order?",
    "When does express shipping arrive?",
    "Does the warranty cover water damage?",
]:
    langfuse.create_dataset_item(
        dataset_name="rag-faithfulness-demo",
        input={"question": question},
        # No expected_output: faithfulness is reference-free
    )

# --- The RAG task under evaluation ---

def rag_task(*, item, **kwargs):
    question = item.input["question"]
    contexts = retrieve(question)
    context_block = "\n\n".join(contexts)
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {
                "role": "system",
                "content": "Answer using only the provided context. If the "
                "context is insufficient, say you do not know.",
            },
            {
                "role": "user",
                "content": f"Context:\n{context_block}\n\nQuestion: {question}",
            },
        ],
    )
    # Return the contexts alongside the answer so evaluators can verify against them
    return {
        "answer": response.choices[0].message.content,
        "contexts": contexts,
    }

# --- The faithfulness judge ---

JUDGE_PROMPT = """You are verifying whether an answer is faithful to its source context.

Context:
{context}

Answer:
{answer}

Extract every factual claim the answer makes. Hedges, questions, and statements
of inability ("I don't know") are not claims. For each claim, decide whether the
context supports it. A claim is supported only if the context states or directly
implies it; paraphrase counts, outside knowledge does not, even if the claim is true.

Respond with JSON only, marking each claim as supported true or false:
{{"claims": [{{"claim": "<claim text>", "supported": true}}]}}"""


def faithfulness(*, input, output, **kwargs):
    context = "\n\n".join(output["contexts"])
    response = client.chat.completions.create(
        model="gpt-4.1",
        messages=[
            {
                "role": "user",
                "content": JUDGE_PROMPT.format(
                    context=context, answer=output["answer"]
                ),
            }
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )
    claims = json.loads(response.choices[0].message.content)["claims"]
    if not claims:
        return Evaluation(
            name="faithfulness", value=1.0, comment="No factual claims made"
        )
    unsupported = [c["claim"] for c in claims if not c["supported"]]
    return Evaluation(
        name="faithfulness",
        value=1 - len(unsupported) / len(claims),
        comment=(
            f"Unsupported claims: {unsupported}"
            if unsupported
            else f"All {len(claims)} claims supported"
        ),
    )


def avg_faithfulness(*, item_results, **kwargs):
    values = [
        e.value
        for result in item_results
        for e in result.evaluations
        if e.name == "faithfulness" and isinstance(e.value, (int, float))
    ]
    return Evaluation(
        name="avg_faithfulness",
        value=sum(values) / len(values) if values else None,
    )

# --- Run the experiment ---

dataset = langfuse.get_dataset("rag-faithfulness-demo")

result = dataset.run_experiment(
    name="Faithfulness baseline",
    description="Claim-level faithfulness judge on the RAG demo dataset",
    task=rag_task,
    evaluators=[faithfulness],
    run_evaluators=[avg_faithfulness],
)

print(result.format())

Every task execution is traced, item scores attach to the item's trace, and avg_faithfulness attaches to the dataset run, so consecutive runs are comparable in the Langfuse UI. From here the useful moves are comparing runs before and after a retrieval or prompt change, and turning the run-level average into a CI gate that fails a pull request when faithfulness regresses.

Score production traces with a managed LLM-as-a-judge evaluator

Offline experiments cover the inputs you thought to test. The same faithfulness judgment can run on live traffic through Langfuse's managed LLM-as-a-judge evaluators, which execute server-side against production observations, without an evaluation job in your own infrastructure:

1. Instrument the context onto the target observation. Observation-level evaluators read only the data on the observation they match; they do not load sibling observations from the trace. Record the retrieved documents on the generation (or root) observation, in its input or metadata, so the judge has something to verify against.
2. Create the evaluator. On the Evaluators page, pick a managed evaluator from the catalog (maintained by Langfuse and partners including Ragas) or define a custom one with your own faithfulness prompt using {{context}}, {{answer}} style variables. The judge model is set through an LLM connection and must support structured output.
3. Target live observations and sample. Filter to the observations that carry your RAG generations, then set a sampling percentage. Judging 5 to 10 percent of traffic is typically enough to see a faithfulness trend without paying to judge everything.
4. Map variables. Map observation fields onto the prompt variables, with JSONPath expressions for nested data. Langfuse previews the populated judge prompt against your recent production data before you save.

Scores land on the matched observations asynchronously, after the response has shipped. That makes online faithfulness scoring a monitoring and triage tool: alert when the average drops, route low-scoring traces to review, and add them to the experiment dataset above so the next release is tested against last week's failures. A check that must block an unfaithful answer before the user sees it belongs in your application as a guardrail, not in an evaluator.

Faithfulness evaluation without new infrastructure

If Langfuse already traces your RAG pipeline, everything on this page runs against the deployment you have. The experiment runner is part of the same Python SDK that does the tracing, authenticates with the same API keys, and writes scores next to the traces and datasets you already accumulate. Managed evaluators run inside the same Langfuse instance. There is no separate evaluation service to deploy, no second vendor account, and no additional data pipeline to operate. Langfuse is open source and can be self-hosted, so judge inputs and scores can stay entirely within your infrastructure when data residency requires it.

Ragas faithfulness and when to use it

Ragas is an established open-source library for RAG evaluation, and its faithfulness metric is a well-known reference implementation of the claim-decomposition approach described above. If your team already evaluates with Ragas, there is no need to rewrite those metrics: the Ragas integration shows how to run Ragas evaluations and attach the resulting scores to Langfuse traces, so Ragas faithfulness numbers land in the same score timeline as everything else.

The native route on this page is the better fit when you want the judge prompt to be visible and editable in the same platform as your traces, when you want managed evaluators sampling production traffic without operating a separate evaluation job, or when faithfulness is one of several metrics (relevance, correctness, custom policy checks) that should share one dataset, one experiment run, and one score table. The two approaches also combine: Ragas offline during development, a managed judge online in production.

FAQ

What is the difference between RAG faithfulness and hallucination detection?

Faithfulness is scoped to RAG: it checks the answer against the specific context retrieved for that request, and an answer that is true but unsupported by the context still fails. Hallucination detection is the broader failure class: fabricated content in any LLM output, with or without retrieval, checked against whatever source of truth applies. In a RAG system, a faithfulness score is usually the most actionable hallucination signal because the comparison target is documents you control. For the broader problem, see hallucination detection.

Does faithfulness evaluation require ground-truth answers?

No. Faithfulness compares the answer to the retrieved context, both of which exist for every request, so it is reference-free. This is why the dataset in the example above has no expected_output and why faithfulness can run on sampled production traffic. Reference-based metrics like answer correctness require a ground-truth answer per item and only run where you have labeled data; a complete RAG evaluation usually combines both kinds.

Which model should I use as the faithfulness judge?

Use a model that is strong at instruction following and supports structured output, and prefer a stronger model than the one generating the answers, since claim verification against long contexts is a demanding reading task. Pin temperature to 0 for repeatability. Calibrate once by having a person label 20 to 50 answers and comparing against the judge's verdicts; if agreement is low, fix the rubric wording before swapping models, because rubric ambiguity is the more common cause.

Should faithfulness be a binary or a graded score?

Make the individual judge decisions binary and let the numeric score emerge from aggregation. Each claim is either supported by the context or it is not, and binary decisions are where LLM judges are most consistent. The aggregated share of supported claims gives you a 0-to-1 metric for dashboards and thresholds, without asking the judge to produce a graded rating whose steps are undefined.

Do I need Ragas to evaluate faithfulness?

No. Faithfulness is a metric, not a library feature: any LLM judge with a claim-extraction prompt implements it, as the experiment code on this page does with the Langfuse SDK alone. Ragas provides a maintained implementation of that judge along with other RAG metrics, and its scores can be pushed into Langfuse through the integration. Choose based on where you want the judge to live: in your own prompt under version control, or in a library you import.