---
title: "Langfuse Integration with Cleanlab"
sidebarTitle: Cleanlab
description: "Automatically evaluate LLMs in real time with Cleanlab's trustworthy Language Model (TLM)"
logo: /images/integrations/cleanlab_icon.png
logoAppearance: dark
---

# Automated Evaluations with Cleanlab

Cleanlab’s [Trustworthy Language Model](https://cleanlab.ai/tlm/) (TLM) enables Langfuse users to quickly identify low quality and hallucinated responses from any LLM trace.

## What is TLM?

TLM is an automated evaluation tool that add reliability and explainability to every LLM output. TLM automatically finds the poor quality and incorrect LLM responses lurking within your production logs and traces. This helps you perform better Evals, with significantly less manual review and annotation work to find these bad responses yourself. TLM also enables smart-routing for LLM-automated responses and decision-making using trustworthiness scores for every LLM output.

**TLM provides users with:**

- Trustworthiness scores and explanation for every LLM response
- Higher accuracy: rigorous [benchmarks](https://cleanlab.ai/blog/trustworthy-language-model/) show TLM consistently produces more accurate results than other LLMs like GPT 4/4o and Claude.
- Scalable API: designed to handle large datasets, TLM is suitable for most enterprise applications, including data extraction, tagging/labeling, Q&A (RAG), and more.

## Getting Started

This guide will walk you through the process of evaluating LLM responses captured in Langfuse with Cleanlab's Trustworthy Language Models (TLM).

### Install dependencies & Set environment variables

```python
%pip install langfuse openai cleanlab-tlm --upgrade
```

```python
import os
import pandas as pd
from getpass import getpass
import dotenv
dotenv.load_dotenv()
```

### API Keys

This guide requires a Cleanlab TLM API key. If you don't have one, you can sign up for a free trial [here](https://tlm.cleanlab.ai/).

This guide requires four API keys:

- [Langfuse Public Key](https://cloud.langfuse.com/) (available in any Langfuse Cloud region: [EU](https://cloud.langfuse.com), [US](https://us.cloud.langfuse.com), [Japan](https://jp.cloud.langfuse.com), or [HIPAA](https://hipaa.cloud.langfuse.com))
- Langfuse Secret Key (from the same project)
- [OpenAI API Key](https://platform.openai.com/api-keys)
- [Cleanlab TLM API Key](https://tlm.cleanlab.ai/)

```python
# Get keys for your project from the project settings page: https://cloud.langfuse.com

os.environ.setdefault("LANGFUSE_PUBLIC_KEY", "pk-lf-...");
os.environ.setdefault("LANGFUSE_SECRET_KEY", "sk-lf-...");
os.environ.setdefault("LANGFUSE_BASE_URL", "https://cloud.langfuse.com"); # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

os.environ.setdefault("OPENAI_API_KEY", "<openai_api_key>");

os.environ.setdefault("CLEANLAB_TLM_API_KEY", "<cleanlab_tlm_api_key>");
```

### Prepare trace dataset and load into Langfuse

For the sake of demonstration purposes, we'll briefly generate some traces and track them in Langfuse. Typically, you would have already captured traces in Langfuse and would skip to "Download trace dataset from Langfuse"

NOTE: TLM requires the entire input to the LLM to be provided. This includes any system prompts, context, or other information that was originally provided to the LLM to generate the response. Notice below that we include the system prompt in the trace metadata since by default the trace does not include the system prompt within the input.

```python
from langfuse import observe, get_client, propagate_attributes
from openai import OpenAI

langfuse = get_client()
openai = OpenAI()
```

```python
# Let's use some tricky trivia questions to generate some traces
trivia_questions = [
    "What is the 3rd month of the year in alphabetical order?",
    "What is the capital of France?",
    "How many seconds are in 100 years?",
    "Alice, Bob, and Charlie went to a café. Alice paid twice as much as Bob, and Bob paid three times as much as Charlie. If the total bill was $72, how much did each person pay?",
    "When was the Declaration of Independence signed?"
]

@observe()
def generate_answers(trivia_question):
    system_prompt = "You are a trivia master."

    # Propagate the trace name, tags, and metadata to all observations of this trace
    with propagate_attributes(
        trace_name=f"Answering question: '{trivia_question}'",
        tags=["TLM_eval_pipeline"],
        metadata={"system_prompt": system_prompt}
    ):
        response = openai.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": trivia_question},
            ],
        )

        answer = response.choices[0].message.content
        return answer


# Generate answers
answers = []
for i in range(len(trivia_questions)):
    answer = generate_answers(trivia_questions[i])
    answers.append(answer)
    print(f"Question {i+1}: {trivia_questions[i]}")
    print(f"Answer {i+1}:\n{answer}\n")

# Ensure all traces are sent to Langfuse before we fetch them below
langfuse.flush()

print(f"Generated {len(answers)} answers and tracked them in Langfuse.")
```

Remember, the goal of this tutorial is to show you how to build an external evaluation pipeline. These pipelines will run in your CI/CD environment, or be run in a different orchestrated container service. No matter the environment you choose, three key steps always apply:

1.  **Fetch Your Traces**: Get your application traces to your evaluation environment
2.  **Run Your Evaluations**: Apply any evaluation logic you prefer
3.  **Save Your Results**: Attach your evaluations back to the Langfuse trace used for calculating them.

For the rest of the notebook, we'll have one goal:

---

🎯 Goal: **_Evaluate all traces run in the past 24 hours_**

---

### Download trace dataset from Langfuse

Fetching recent root observations from Langfuse is straightforward. We'll use the Observations API v2 via the SDK, filter to logical root observations, and then evaluate that sample. After that, we'll add our scores back into Langfuse.

The `langfuse.api.observations.get_many()` method supports timestamp filters, field selection, and structured filters for trace tags. You can find more querying patterns in our [query-via-sdk docs](https://langfuse.com/docs/api-and-data-platform/features/query-via-sdk).

```python
from langfuse import get_client
from datetime import datetime, timedelta
import json

langfuse = get_client()
now = datetime.now()
one_day_ago = now - timedelta(hours=24)

# The advanced `filter` parameter takes precedence over the individual query
# parameters, so the time window is expressed as filter conditions as well.
traces = langfuse.api.observations.get_many(
    fields="core,basic,io,metadata,trace_context",
    filter=json.dumps([
        {"type": "boolean", "column": "isRootObservation", "operator": "=", "value": True},
        {"type": "arrayOptions", "column": "tags", "operator": "all of", "value": ["TLM_eval_pipeline"]},
        {"type": "datetime", "column": "startTime", "operator": ">=", "value": one_day_ago.isoformat()},
        {"type": "datetime", "column": "startTime", "operator": "<", "value": now.isoformat()},
    ]),
).data
```

### Generate evaluations with TLM

Langfuse can handle numerical, boolean and categorical (`string`) scores. Wrapping your custom evaluation logic in a function is often a good practice.

Instead of running TLM individually on each trace, we'll provide all of the prompt, response pairs in a list to TLM in a single call. This is more efficient and allows us to get scores and explanations for all of the traces at once. Then, using the `trace.id`, we can attach the scores and explanations back to the correct trace in Langfuse.

```python
from cleanlab_tlm import TLM

tlm = TLM(options={"log": ["explanation"]})
```

```python
# This helper extracts prompts and responses from the root observations.
def parse_json_if_needed(value):
    if isinstance(value, str):
        try:
            return json.loads(value)
        except json.JSONDecodeError:
            return value
    return value


def get_prompt_response_pairs(traces):
    prompts = []
    responses = []
    for trace in traces:
        metadata = parse_json_if_needed(trace.metadata)
        trace_input = parse_json_if_needed(trace.input)
        prompt = trace_input["args"][0] if isinstance(trace_input, dict) else trace_input
        prompts.append(metadata["system_prompt"] + "\n" + prompt)
        responses.append(parse_json_if_needed(trace.output))
    return prompts, responses

trace_ids = [trace.trace_id for trace in traces]
prompts, responses = get_prompt_response_pairs(traces)
```

Now, let's use TLM to generate a `trustworthiness score` and `explanation` for each trace.

**IMPORTANT:** It is essential to always include any system prompts, context, or other information that was originally provided to the LLM to generate the response. You should construct the prompt input to `get_trustworthiness_score()` in a way that is as similar as possible to the original prompt. This is why we included the system prompt in the trace metadata.

```python
# Evaluate each of the prompt, response pairs using TLM
evaluations = tlm.get_trustworthiness_score(prompts, responses)

# Extract the trustworthiness scores and explanations from the evaluations
trust_scores = [entry["trustworthiness_score"] for entry in evaluations]
explanations = [entry["log"]["explanation"] for entry in evaluations]

# Create a DataFrame with the evaluation results
trace_evaluations = pd.DataFrame({
    'trace_id': trace_ids,
    'prompt': prompts,
    'response': responses,
    'trust_score': trust_scores,
    'explanation': explanations
})
trace_evaluations
```

    Querying TLM... 100%|██████████|

<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }

</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>trace_id</th>
      <th>prompt</th>
      <th>response</th>
      <th>trust_score</th>
      <th>explanation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>2f0d41b2-9b89-4ba6-8b3f-7dadac8a8fae</td>
      <td>You are a trivia master.\nWhen was the Declara...</td>
      <td>The Declaration of Independence was signed on ...</td>
      <td>0.389889</td>
      <td>The proposed response states that the Declarat...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>f8e91744-3fcb-4ef5-b6c6-7cbcf0773144</td>
      <td>You are a trivia master.\nAlice, Bob, and Char...</td>
      <td>Let's denote the amount Charlie paid as C. \n\...</td>
      <td>0.669774</td>
      <td>This response is untrustworthy due to lack of ...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>f9b42125-4e5e-4533-bfbb-36c30490bd1d</td>
      <td>You are a trivia master.\nHow many seconds are...</td>
      <td>There are 3,153,600,000 seconds in 100 years.</td>
      <td>0.499818</td>
      <td>To calculate the number of seconds in 100 year...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>71b131b9-e706-41c7-9bfd-b77719783f29</td>
      <td>You are a trivia master.\nWhat is the capital ...</td>
      <td>The capital of France is Paris.</td>
      <td>0.987433</td>
      <td>Did not find a reason to doubt trustworthiness.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>da0ee9fa-01cf-42ce-9e3e-e8d127ca105b</td>
      <td>You are a trivia master.\nWhat is the 3rd mont...</td>
      <td>March.</td>
      <td>0.114874</td>
      <td>To determine the 3rd month of the year in alph...</td>
    </tr>
  </tbody>
</table>

Awesome! Now we have a DataFrame mapping trace IDs to their scores and explanations. We've also included the prompt and response for each trace for demonstration purposes to find the **least trustworthy trace!**

```python
sorted_df = trace_evaluations.sort_values(by="trust_score", ascending=True).head()
sorted_df
```

<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }

</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>trace_id</th>
      <th>prompt</th>
      <th>response</th>
      <th>trust_score</th>
      <th>explanation</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>4</th>
      <td>da0ee9fa-01cf-42ce-9e3e-e8d127ca105b</td>
      <td>You are a trivia master.\nWhat is the 3rd mont...</td>
      <td>March.</td>
      <td>0.114874</td>
      <td>To determine the 3rd month of the year in alph...</td>
    </tr>
    <tr>
      <th>0</th>
      <td>2f0d41b2-9b89-4ba6-8b3f-7dadac8a8fae</td>
      <td>You are a trivia master.\nWhen was the Declara...</td>
      <td>The Declaration of Independence was signed on ...</td>
      <td>0.389889</td>
      <td>The proposed response states that the Declarat...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>f9b42125-4e5e-4533-bfbb-36c30490bd1d</td>
      <td>You are a trivia master.\nHow many seconds are...</td>
      <td>There are 3,153,600,000 seconds in 100 years.</td>
      <td>0.499818</td>
      <td>To calculate the number of seconds in 100 year...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>f8e91744-3fcb-4ef5-b6c6-7cbcf0773144</td>
      <td>You are a trivia master.\nAlice, Bob, and Char...</td>
      <td>Let's denote the amount Charlie paid as C. \n\...</td>
      <td>0.669774</td>
      <td>This response is untrustworthy due to lack of ...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>71b131b9-e706-41c7-9bfd-b77719783f29</td>
      <td>You are a trivia master.\nWhat is the capital ...</td>
      <td>The capital of France is Paris.</td>
      <td>0.987433</td>
      <td>Did not find a reason to doubt trustworthiness.</td>
    </tr>
  </tbody>
</table>

```python
# Let's look at the least trustworthy trace.
print("Prompt: ", sorted_df.iloc[0]["prompt"], "\n")
print("OpenAI Response: ", sorted_df.iloc[0]["response"], "\n")
print("TLM Trust Score: ", sorted_df.iloc[0]["trust_score"], "\n")
print("TLM Explanation: ", sorted_df.iloc[0]["explanation"])
```

    Prompt:  You are a trivia master.
    What is the 3rd month of the year in alphabetical order?

    OpenAI Response:  March.

    TLM Trust Score:  0.11487442493072615

    TLM Explanation:  To determine the 3rd month of the year in alphabetical order, we first list the months: January, February, March, April, May, June, July, August, September, October, November, December. When we arrange these months alphabetically, we get: April, August, December, February, January, July, June, March, May, November, October, September. In this alphabetical list, March is the 8th month, not the 3rd. The 3rd month in alphabetical order is actually December. Therefore, the proposed response is incorrect.
    This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either):
    December.

#### Awesome! TLM was able to identify multiple traces that contained incorrect answers from OpenAI.

Let's upload the `trust_score` and `explanation` columns to Langfuse.

### Upload evaluations to Langfuse

```python
for idx, row in trace_evaluations.iterrows():
    trace_id = row["trace_id"]
    trust_score = row["trust_score"]
    explanation = row["explanation"]

    # Add the trustworthiness score to the trace with the explanation as a comment
    langfuse.create_score(
        trace_id=trace_id,
        name="trust_score",
        value=trust_score,
        comment=explanation
    )

langfuse.flush()
```

You should now see the TLM trustworthiness score and explanation in the Langfuse UI!

![Image of Langfuse platform showing Cleanlab's TLM trust score](https://langfuse.com/images/cookbook/integration-cleanlab/tlm_trust_scores.png)

If you click on a trace, you can also see the trust score and provided explanation.

![Image of Langfuse platform showing Cleanlab's TLM trust score and explanation](https://langfuse.com/images/cookbook/integration-cleanlab/tlm_trust_scores_explanation.png)

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/integrations/model-providers/cleanlab.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.