Promptfoo Integration with Langfuse Prompt Management

Promptfoo is an open-source LLM testing platform. It offers tools for red teaming, vulnerability scanning, and LLM evaluations, helping teams ensure the security and reliability of their LLM applications.

Integrate Promptfoo with Langfuse to take advantage of Langfuse’s prompt management features during your Promptfoo evaluations. By using this integration, you can easily reference and manage prompts directly within Langfuse, making your LLM testing process more efficient and organized.

With Langfuse Prompt Management, you can:

Update prompts without redeploying your application.
Track and revert to previous prompt versions.
Monitor and optimize prompt performance.
Integrate prompts seamlessly with your tools and applications.
Manage prompts via UI, SDKs, or API with minimal latency.

For more details, visit the Langfuse Prompt Management documentation.

Thanks to the team at Promptfoo for developing this integration (docs).

Quick Start Guide

Step 1: Set up Langfuse

Install the langfuse SDK: npm install langfuse
Visit Langfuse to create an account.
Create a new project and copy your Langfuse API keys.
Set the LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST environment variables as desired.

Step 2: Create a Langfuse Prompt

You can create, edit and manage your prompts via the Langfuse UI, SDKs, or API.

# Create a text prompt
langfuse.create_prompt(
    name="movie-critic",
    type="text",
    prompt="As a {{criticlevel}} movie critic, do you like {{movie}}?",
    labels=["production"],  # directly promote to production
    config={
        "model": "gpt-3.5-turbo",
        "temperature": 0.7,
        "supported_languages": ["en", "fr"],
    },  # optionally, add configs (e.g. model parameters or model tools) or tags
)
 
# Create a chat prompt
langfuse.create_prompt(
    name="movie-critic-chat",
    type="chat",
    prompt=[
      { role: "system", content: "You are an {{criticlevel}} movie critic" },
      { role: "user", content: "Do you like {{movie}}?" },
    ],
    labels=["production"],  # directly promote to production
    config={
        "model": "gpt-3.5-turbo",
        "temperature": 0.7,
        "supported_languages": ["en", "fr"],
    },  # optionally, add configs (e.g. model parameters or model tools) or tags
)

If you already have a prompt with the same name, the prompt will be added as a new version.

// Create a text prompt
await langfuse.createPrompt({
  name: "movie-critic",
  type: "text",
  prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
  labels: ["production"], // directly promote to production
  config: {
    model: "gpt-3.5-turbo",
    temperature: 0.7,
    supported_languages: ["en", "fr"],
  }, // optionally, add configs (e.g. model parameters or model tools) or tags
});
 
// Create a chat prompt
await langfuse.createPrompt({
  name: "movie-critic-chat",
  type: "chat",
  prompt: [
    { role: "system", content: "You are an {{criticlevel}} movie critic" },
    { role: "user", content: "Do you like {{movie}}?" },
  ],
  labels: ["production"], // directly promote to production
  config: {
    model: "gpt-3.5-turbo",
    temperature: 0.7,
    supported_languages: ["en", "fr"],
  }, // optionally, add configs (e.g. model parameters or model tools) or tags
});

If you already have a prompt with the same name, the prompt will be added as a new version.

Step 3: Reference Prompts in Promptfoo

Now you can use the prompts you created in Langfuse with Promptfoo. Use the langfuse:// prefix for your prompts in your Promptfoo configuration file, followed by the Langfuse prompt ID and version. For example:

prompts:
  - "langfuse://foo-bar-prompt:3"
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      # ...

Variables from your Promptfoo test cases will be automatically plugged into the Langfuse prompt as variables.

Promptfoo Integration with Langfuse Prompt Management

Quick Start Guide

Step 1: Set up Langfuse

Step 2: Create a Langfuse Prompt

Step 3: Reference Prompts in Promptfoo

GitHub Discussions

Was this page useful?

Questions? We're here to help

Subscribe to updates