Automated Evaluations with Cleanlab

%pip install -q langfuse openai cleanlab-tlm --upgrade

import os
import pandas as pd
from getpass import getpass
import dotenv
dotenv.load_dotenv()

# Get keys for your project from the project settings page: https://cloud.langfuse.com
 
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..." 
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region
 
os.environ["OPENAI_API_KEY"] = "<openai_api_key>"
 
os.environ["CLEANLAB_TLM_API_KEY"] = "<cleanlab_tlm_api_key>"

from langfuse.decorators import langfuse_context, observe
from openai import OpenAI
 
openai = OpenAI()

# Let's use some tricky trivia questions to generate some traces
trivia_questions = [    
    "What is the 3rd month of the year in alphabetical order?",
    "What is the capital of France?",
    "How many seconds are in 100 years?",
    "Alice, Bob, and Charlie went to a café. Alice paid twice as much as Bob, and Bob paid three times as much as Charlie. If the total bill was $72, how much did each person pay?",
    "When was the Declaration of Independence signed?"
]
 
@observe()
def generate_answers(trivia_question):
    system_prompt = "You are a trivia master."
 
    # Update the trace with the question    
    langfuse_context.update_current_trace(
        name=f"Answering question: '{trivia_question}'",
        tags=["TLM_eval_pipeline"],
        metadata={"system_prompt": system_prompt}
    )
 
    response = openai.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": trivia_question},
        ],
    )
    
    answer = response.choices[0].message.content
    return answer
 
 
# Generate answers
answers = []
for i in range(len(trivia_questions)):
    answer = generate_answers(trivia_questions[i])
    answers.append(answer)  
    print(f"Question {i+1}: {trivia_questions[i]}")
    print(f"Answer {i+1}:\n{answer}\n")
 
print(f"Generated {len(answers)} answers and tracked them in Langfuse.")

Question 1: What is the 3rd month of the year in alphabetical order?
Answer 1:
March

Question 2: What is the capital of France?
Answer 2:
The capital of France is Paris.

Question 3: How many seconds are in 100 years?
Answer 3:
There are 31,536,000 seconds in a year (60 seconds x 60 minutes x 24 hours x 365 days). Therefore, in 100 years, there would be 3,153,600,000 seconds.

Question 4: Alice, Bob, and Charlie went to a café. Alice paid twice as much as Bob, and Bob paid three times as much as Charlie. If the total bill was $72, how much did each person pay?
Answer 4:
Let's call the amount Charlie paid x. 
Alice paid twice as much as Bob, so she paid 2*(3x) = 6x.
Bob paid three times as much as Charlie, so he paid 3x.

We know the total bill was $72:
x + 6x + 3x = 72
10x = 72
x = 7.2 

Therefore, Charlie paid $7.20, Bob paid $21.60, and Alice paid $43.20.

Question 5: When was the Declaration of Independence signed?
Answer 5:
The Declaration of Independence was signed on July 4, 1776.

Generated 5 answers and tracked them in Langfuse.

from langfuse import Langfuse
from datetime import datetime, timedelta
 
langfuse = Langfuse()
now = datetime.now()
one_day_ago = now - timedelta(hours=24)
 
traces = langfuse.fetch_traces(
    tags="TLM_eval_pipeline",
    from_timestamp=one_day_ago,
    to_timestamp=now,
).data
 

from cleanlab_tlm import TLM
 
tlm = TLM(options={"log": ["explanation"]})

# This helper method will extract the prompt and response from each trace and return three lists: trace ID's, prompts, and responses.
def get_prompt_response_pairs(traces):
    prompts = []
    responses = []
    for trace in traces:
        prompts.append(trace.metadata["system_prompt"] + "\n" + trace.input["args"][0])
        responses.append(trace.output)
    return prompts, responses
 
trace_ids = [trace.id for trace in traces]
prompts, responses = get_prompt_response_pairs(traces)

# Evaluate each of the prompt, response pairs using TLM
evaluations = tlm.get_trustworthiness_score(prompts, responses)
 
# Extract the trustworthiness scores and explanations from the evaluations
trust_scores = [entry["trustworthiness_score"] for entry in evaluations]
explanations = [entry["log"]["explanation"] for entry in evaluations]
 
# Create a DataFrame with the evaluation results
trace_evaluations = pd.DataFrame({
    'trace_id': trace_ids,
    'prompt': prompts,
    'response': responses, 
    'trust_score': trust_scores,
    'explanation': explanations
})
trace_evaluations

Querying TLM... 100%|██████████|

sorted_df = trace_evaluations.sort_values(by="trust_score", ascending=True).head()
sorted_df

# Let's look at the least trustworthy trace.
print("Prompt: ", sorted_df.iloc[0]["prompt"], "\n")
print("OpenAI Response: ", sorted_df.iloc[0]["response"], "\n")
print("TLM Trust Score: ", sorted_df.iloc[0]["trust_score"], "\n")
print("TLM Explanation: ", sorted_df.iloc[0]["explanation"])

Prompt:  You are a trivia master.
What is the 3rd month of the year in alphabetical order? 

OpenAI Response:  March. 

TLM Trust Score:  0.11487442493072615 

TLM Explanation:  To determine the 3rd month of the year in alphabetical order, we first list the months: January, February, March, April, May, June, July, August, September, October, November, December. When we arrange these months alphabetically, we get: April, August, December, February, January, July, June, March, May, November, October, September. In this alphabetical list, March is the 8th month, not the 3rd. The 3rd month in alphabetical order is actually December. Therefore, the proposed response is incorrect. 
This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either): 
December.

for idx, row in trace_evaluations.iterrows():
    trace_id = row["trace_id"]
    trust_score = row["trust_score"]
    explanation = row["explanation"]
    
    # Add the trustworthiness score to the trace with the explanation as a comment
    langfuse.score(
            trace_id=trace_id,
            name="trust_score",
            value=trust_score,
            comment=explanation
        )