Migrate a Promptfoo eval suite to Langfuse

This guide walks through moving an existing Promptfoo eval suite to Langfuse datasets and experiments: test cases first, then assertions, prompts, the provider matrix, and the CI gate.

TL;DR: A promptfooconfig.yaml maps cleanly onto Langfuse concepts. tests and their vars become dataset items, deterministic assertions become code evaluators, model-graded assertions become LLM-as-a-judge evaluators, and promptfoo eval in CI becomes an experiment script that raises RegressionError inside the langfuse/experiment-action GitHub Action. What changes structurally: Promptfoo evals are declarative YAML runs whose results live in local files or a shared viewer, while Langfuse experiments are code that runs against datasets on a platform where your production traces, scores, and prompts already live. You can migrate incrementally and keep Promptfoo running side-by-side, and Promptfoo's red teaming has no Langfuse equivalent, so keep it for that.

Why teams migrate

Promptfoo is a good CI harness for prompt testing: a declarative config, a broad assertion library, and a fast local loop with promptfoo eval and promptfoo view. Teams typically outgrow it in one specific way: eval results live apart from production. Promptfoo runs produce output files and a results viewer, with sharing to promptfoo.app or a self-hosted sharing server (as of July 2026, per the Promptfoo sharing docs). Production telemetry lives somewhere else entirely.

Moving the suite into Langfuse changes that:

• Eval results land next to production traces. Every experiment item execution creates a full trace, so a failing test case is inspectable with the same tooling you use for production incidents, and the same datasets can be built from real production traces.
• Non-engineers can see and shape evals. Datasets, experiment comparisons, and annotation queues are in the Langfuse UI, so product managers and domain experts review outputs and edit test cases without checking out a repo.
• Judges run as managed platform evaluators. Langfuse ships a catalog of managed LLM-as-a-judge evaluators, including prebuilt evaluators developed in partnership with Ragas, and the same judges score both offline runs and live production traffic.
• The whole platform is self-hostable. Both tools are open source with MIT-licensed cores; the difference is what you can run. Self-hosting Langfuse gives you the full platform (UI, API, storage) in your own infrastructure as the persistent system of record for eval history.

Be equally clear about what you give up. Promptfoo's declarative assertion catalog is broad (30+ types including rouge-n, bleu, perplexity, latency, and cost, all negatable and weightable); in Langfuse, deterministic checks are code you write as evaluator functions or code evaluators, with the autoevals library covering many model-graded and heuristic checks out of the box. The zero-code YAML workflow becomes a Python or TypeScript script. And Promptfoo's provider matrix, which expands providers × prompts × tests automatically, becomes a loop you write yourself. If the declarative harness is all you need and results-in-files is acceptable, Promptfoo remains a reasonable choice; this guide is for teams consolidating evals where their traces live.

Concept mapping

Supported data types

The example suite

The steps below migrate this small but representative config: two test cases, a deterministic assertion each, and a model-graded rubric applied to every test via defaultTest.

# promptfooconfig.yaml (before)
prompts:
  - file://prompts/support-agent.txt
providers:
  - openai:gpt-4.1
defaultTest:
  assert:
    - type: llm-rubric
      value: "Is polite and offers a concrete next step"
tests:
  - vars:
      question: "How do I reset my password?"
    assert:
      - type: icontains
        value: "reset link"
  - vars:
      question: "What is your refund window?"
    assert:
      - type: icontains
        value: "30 days"

Snippets below show both the Python SDK v4 (pip install langfuse) and the JS/TS SDK v5 (npm install @langfuse/client). Both read LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL from the environment.

Step 1: Turn tests into a dataset

Each entry in tests becomes a dataset item. The vars map to input, and per-test reference values (here, the icontains string) move to expected_output. Deriving the item ID from the test content makes the import idempotent: re-running the script updates items in place instead of duplicating them.

import hashlib
import json

from langfuse import get_client

langfuse = get_client()

langfuse.create_dataset(name="support-agent-regression")

test_cases = [
    {"question": "How do I reset my password?", "must_contain": "reset link"},
    {"question": "What is your refund window?", "must_contain": "30 days"},
]

for case in test_cases:
    langfuse.create_dataset_item(
        dataset_name="support-agent-regression",
        # deterministic ID = stable retry key, so re-runs don't duplicate items
        id=hashlib.sha256(json.dumps(case, sort_keys=True).encode()).hexdigest()[:16],
        input={"question": case["question"]},
        expected_output=case["must_contain"],
    )

import { createHash } from "node:crypto";
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();

await langfuse.dataset.create("support-agent-regression");

const testCases = [
  { question: "How do I reset my password?", mustContain: "reset link" },
  { question: "What is your refund window?", mustContain: "30 days" },
];

for (const testCase of testCases) {
  await langfuse.dataset.createItem({
    datasetName: "support-agent-regression",
    // deterministic ID = stable retry key, so re-runs don't duplicate items
    id: createHash("sha256")
      .update(JSON.stringify(testCase))
      .digest("hex")
      .slice(0, 16),
    input: { question: testCase.question },
    expectedOutput: testCase.mustContain,
  });
}

Suites that load tests from files (tests: file://tests.csv) export the same way: each CSV column is a variable, and Promptfoo's special __expected column(s) carry the assertion values, so map columns to input and the __expected value (minus its type prefix, e.g. contains:) to expected_output:

import csv

with open("tests.csv") as f:
    for row in csv.DictReader(f):
        expected = row.pop("__expected", None)  # e.g. "contains: reset link"
        # row -> input vars; expected (minus the type prefix) -> expected_output

Confirm in the Langfuse UI that the dataset's item count matches the number of test cases in your config and that expected_output is populated on each item.

Unlike YAML tests in a repo, dataset items are editable in the UI, versioned, and can be created directly from interesting production traces, which is how regression suites usually grow after the migration.

Step 2: Port assertions to evaluators

Evaluators are plain functions that receive the item's input, output, and expected_output and return an Evaluation. The icontains assertion becomes a few lines of code, and the llm-rubric assertion becomes a judge function that calls a model. Like defaultTest, evaluators passed to an experiment apply to every item.

from langfuse import Evaluation
from langfuse.openai import OpenAI  # traced drop-in replacement

def contains_expected(*, output, expected_output, **kwargs):
    # replaces: type: icontains
    passed = (expected_output or "").lower() in (output or "").lower()
    return Evaluation(name="contains_expected", value=1.0 if passed else 0.0)

RUBRIC = "Is polite and offers a concrete next step"

def rubric_judge(*, input, output, **kwargs):
    # replaces: type: llm-rubric
    response = OpenAI().chat.completions.create(
        model="gpt-4.1",
        messages=[{
            "role": "user",
            "content": (
                f"Grade this response against the rubric: {RUBRIC}\n\n"
                f"Question: {input['question']}\n\nResponse: {output}\n\n"
                'Reply "PASS" or "FAIL", then a one-sentence reason.'
            ),
        }],
    )
    verdict = response.choices[0].message.content or ""
    return Evaluation(
        name="rubric",
        value=1.0 if verdict.strip().upper().startswith("PASS") else 0.0,
        comment=verdict,
    )

import OpenAI from "openai";
import { observeOpenAI } from "@langfuse/openai"; // traced client wrapper
import type { Evaluation } from "@langfuse/client";

const openai = observeOpenAI(new OpenAI());

async function containsExpected({
  output,
  expectedOutput,
}: {
  output: string;
  expectedOutput?: string;
}): Promise<Evaluation> {
  // replaces: type: icontains
  const passed = (output ?? "")
    .toLowerCase()
    .includes((expectedOutput ?? "").toLowerCase());
  return { name: "contains_expected", value: passed ? 1 : 0 };
}

const RUBRIC = "Is polite and offers a concrete next step";

async function rubricJudge({
  input,
  output,
}: {
  input: { question: string };
  output: string;
}): Promise<Evaluation> {
  // replaces: type: llm-rubric
  const response = await openai.chat.completions.create({
    model: "gpt-4.1",
    messages: [
      {
        role: "user",
        content:
          `Grade this response against the rubric: ${RUBRIC}\n\n` +
          `Question: ${input.question}\n\nResponse: ${output}\n\n` +
          'Reply "PASS" or "FAIL", then a one-sentence reason.',
      },
    ],
  });
  const verdict = response.choices[0].message.content ?? "";
  return {
    name: "rubric",
    value: verdict.trim().toUpperCase().startsWith("PASS") ? 1 : 0,
    comment: verdict,
  };
}

For Promptfoo's other model-graded assertions, you rarely need to write the judge yourself:

• The autoevals library ships Factuality (adapted from OpenAI's evals project, the same lineage as Promptfoo's factuality assertion), ClosedQA, embedding similarity, and RAG metrics covering context precision, recall, and faithfulness. Langfuse converts any of them into an evaluator with create_evaluator_from_autoevals (imported from langfuse.experiment in Python, createEvaluatorFromAutoevals from @langfuse/client in JS/TS; see the experiment SDK docs).
• Langfuse-managed LLM-as-a-judge evaluators are configured in the UI and executed by the platform against experiments or live traffic, so judge prompts stop living in application code entirely.

Deterministic checks you want non-engineers to see and reuse can also be authored directly in the Langfuse UI as code evaluators, which Langfuse executes on experiment observations.

Step 3: Move prompts into Langfuse

The example config references file://prompts/support-agent.txt. Promptfoo templates use the same {{variable}} syntax as Langfuse prompt management, so the file's content ports verbatim; the model from the providers list moves into the prompt's config:

from langfuse import get_client

langfuse = get_client()

with open("prompts/support-agent.txt") as f:
    template = f.read()  # {{question}} works unchanged

langfuse.create_prompt(
    name="support-agent",
    type="text",
    prompt=template,
    config={"model": "gpt-4.1"},  # from the providers list
    labels=["production"],
    commit_message="Migrated from Promptfoo",
)

import { readFileSync } from "node:fs";
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();

const template = readFileSync("prompts/support-agent.txt", "utf8"); // {{question}} works unchanged

await langfuse.prompt.create({
  name: "support-agent",
  type: "text",
  prompt: template,
  config: { model: "gpt-4.1" }, // from the providers list
  labels: ["production"],
  commitMessage: "Migrated from Promptfoo",
});

Confirm in the Langfuse UI that the prompt renders with its variables intact and carries the production label. To compare a prompt variant in the next step, create a second version (edit in the UI or call create_prompt again) and assign it the candidate label.

Moving prompts first also unlocks a useful transition state: Promptfoo natively resolves langfuse:// prompt references, so your existing Promptfoo suite can test the Langfuse-managed prompt while the rest of the migration is in flight (see what you can keep running).

Step 4: Run the experiment (and the provider matrix)

The experiment runner loops your application over the dataset, traces every execution, and applies the evaluators. Where Promptfoo calls the provider for you, the Langfuse task function is your own code, which means the eval exercises your actual retrieval, tools, and glue logic rather than a bare model call.

def support_agent_task(*, item, **kwargs):
    return answer_question(item.input["question"])  # your application logic

dataset = langfuse.get_dataset("support-agent-regression")

result = dataset.run_experiment(
    name="post-migration-baseline",
    task=support_agent_task,
    evaluators=[contains_expected, rubric_judge],
)
print(result.format())

langfuse.flush()  # flush pending spans before a short-lived script exits

import { NodeSDK } from "@opentelemetry/sdk-node";
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { LangfuseClient, ExperimentItem } from "@langfuse/client";

const otelSdk = new NodeSDK({ spanProcessors: [new LangfuseSpanProcessor()] });
otelSdk.start();

const langfuse = new LangfuseClient();
const dataset = await langfuse.dataset.get("support-agent-regression");

const supportAgentTask = async (item: ExperimentItem) =>
  answerQuestion((item.input as { question: string }).question); // your application logic

const result = await dataset.runExperiment({
  name: "post-migration-baseline",
  task: supportAgentTask,
  evaluators: [containsExpected, rubricJudge],
});
console.log(await result.format());

await otelSdk.shutdown(); // flush pending spans before the script exits

Promptfoo's providers and prompts lists expand into a grid automatically. In Langfuse, you express a variant comparison as one experiment run per variant on the same dataset, typically looping over prompt management versions or labels:

from langfuse.openai import OpenAI

for label in ["production", "candidate"]:
    prompt = langfuse.get_prompt("support-agent", label=label)

    def task(*, item, prompt=prompt, **kwargs):
        compiled = prompt.compile(question=item.input["question"])
        response = OpenAI().chat.completions.create(
            model=prompt.config.get("model", "gpt-4.1"),
            messages=[{"role": "user", "content": compiled}],
        )
        return response.choices[0].message.content

    dataset.run_experiment(
        name=f"support-agent ({label})",
        task=task,
        evaluators=[contains_expected, rubric_judge],
    )

for (const label of ["production", "candidate"]) {
  const prompt = await langfuse.prompt.get("support-agent", { label });

  await dataset.runExperiment({
    name: `support-agent (${label})`,
    task: async (item) => {
      const compiled = prompt.compile({
        question: (item.input as { question: string }).question,
      });
      const response = await openai.chat.completions.create({
        model: (prompt.config as { model?: string }).model ?? "gpt-4.1",
        messages: [{ role: "user", content: compiled }],
      });
      return response.choices[0].message.content ?? "";
    },
    evaluators: [containsExpected, rubricJudge],
  });
}

Promptfoo's provider axis works the same way: loop over model names instead of prompt labels, pass the model into the LLM call, and start one run per model. Each run appears side-by-side in the dataset's experiment comparison view, which replaces promptfoo view as the place where variant results are compared, with the difference that every result links to a full trace. Open the comparison view and confirm each run shows scores from both evaluators and every item links to a trace with tokens and cost.

Step 5: Keep failing CI on regressions

promptfoo eval exits with code 100 when a test case fails (as of July 2026, per the Promptfoo CLI docs), and promptfoo/promptfoo-action posts a results comment on pull requests. The Langfuse equivalent is the langfuse/experiment-action GitHub Action: your experiment script raises RegressionError when a metric violates its threshold, the action fails the job, and it posts a PR comment with run-level scores, an item-level table, and a link to the experiment in Langfuse.

# experiments/support-agent-gate.py
# support_agent_task, contains_expected, and rubric_judge as defined in steps 2 and 4
from langfuse import Evaluation, RegressionError, RunnerContext

THRESHOLD = 0.9

def pass_rate(*, item_results, **kwargs):
    scores = [
        e.value
        for item in item_results
        for e in item.evaluations
        if e.name == "contains_expected"
    ]
    return Evaluation(name="pass_rate", value=sum(scores) / len(scores) if scores else 0.0)

def experiment(context: RunnerContext):
    result = context.run_experiment(
        name="PR gate: support agent",
        task=support_agent_task,
        evaluators=[contains_expected, rubric_judge],
        run_evaluators=[pass_rate],
    )

    rate = next(
        (e.value for e in result.run_evaluations if e.name == "pass_rate"), None
    )

    if not isinstance(rate, (int, float)) or rate < THRESHOLD:
        raise RegressionError(
            result=result,
            metric="pass_rate",
            value=float(rate) if isinstance(rate, (int, float)) else 0.0,
            threshold=THRESHOLD,
        )

    return result

// experiments/support-agent-gate.ts
// supportAgentTask, containsExpected, and rubricJudge as defined in steps 2 and 4
import {
  RegressionError,
  type Evaluation,
  type RunnerContext,
} from "@langfuse/client";

const THRESHOLD = 0.9;

async function passRate({
  itemResults,
}: {
  itemResults: Array<{ evaluations: Evaluation[] }>;
}): Promise<Evaluation> {
  const scores = itemResults
    .flatMap((item) => item.evaluations)
    .filter((evaluation) => evaluation.name === "contains_expected")
    .map((evaluation) => Number(evaluation.value));

  return {
    name: "pass_rate",
    value: scores.length
      ? scores.reduce((sum, score) => sum + score, 0) / scores.length
      : 0,
  };
}

export async function experiment(context: RunnerContext) {
  const result = await context.runExperiment({
    name: "PR gate: support agent",
    task: supportAgentTask,
    evaluators: [containsExpected, rubricJudge],
    runEvaluators: [passRate],
  });

  const rate = result.runEvaluations.find(
    (evaluation) => evaluation.name === "pass_rate",
  )?.value;

  if (typeof rate !== "number" || rate < THRESHOLD) {
    throw new RegressionError({
      result,
      metric: "pass_rate",
      value: typeof rate === "number" ? rate : 0,
      threshold: THRESHOLD,
    });
  }

  return result;
}

# .github/workflows/eval-gate.yml (excerpt)
- uses: langfuse/experiment-action@v1.0.10
  with:
    langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
    langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}
    experiment_path: experiments/support-agent-gate.py # or .ts
    dataset_name: support-agent-regression
    github_token: ${{ github.token }}

The action supports pinning a dataset_version for reproducible runs and exposes a normalized result_json output for downstream steps. Verify the gate the same way you would in Promptfoo: break the prompt on a branch and confirm the workflow fails and the PR comment shows the regressed metric. For a deeper treatment of regression gates, thresholds, and flaky-judge handling, see our guide to LLM regression testing.

What you can keep running

Migration does not have to be a cutover, and two Promptfoo workflows remain useful alongside Langfuse:

• Promptfoo evals against Langfuse-managed prompts. Promptfoo natively resolves langfuse://prompt-name@production references in its prompts list, so moving prompts into Langfuse first lets both systems test the same source of truth during the transition. The Promptfoo integration page covers the setup; this migration guide is the optional endpoint of that path.
• Red teaming. Promptfoo's redteam and model-scanning capabilities target security probing, which Langfuse does not do. Teams that migrate their quality evals commonly keep Promptfoo in the toolchain for adversarial testing.

Historical Promptfoo results are best archived rather than imported: scores are cheap to regenerate against the migrated dataset, and re-running your latest suite as a post-migration-baseline experiment gives future runs a clean anchor.

Validation checklist

• Dataset item count matches the number of Promptfoo test cases (including any CSV-loaded tests)
• Every assertion in the old config has a corresponding evaluator (code, autoevals, or managed judge)
• Reference values (contains strings, factuality references) moved into expected_output
• Prompts resolve from Langfuse by name and label, with variables rendering correctly
• Baseline experiment run completed with all evaluators producing scores
• Judge evaluator spot-checked against a handful of known-good and known-bad outputs
• CI gate fails on a deliberately broken prompt (mirror of exit code 100 behavior)
• PR comment from langfuse/experiment-action renders scores and links to the run
• Team members who used promptfoo view have Langfuse project access
• Decision recorded on what stays in Promptfoo (red teaming, side-by-side window end date)

Limitations and gaps

Know these before you commit to the cutover:

• Per-assertion weight and threshold have no direct equivalent. Compute composite or weighted scores in a run-level evaluator instead.
• Statistical assertion types are library calls, not built-ins. rouge-n, bleu, and perplexity need a library call inside an evaluator function.
• latency and cost assertions don't port as assertions. Langfuse records latency, token usage, and cost on every experiment trace, so monitor them in the experiment comparison view or on dashboards rather than as per-item pass/fail checks.
• The declarative YAML workflow becomes code. Test cases stay declarative (dataset items in the UI), but assertions and the run loop are Python or TypeScript.
• Red teaming is not replaced. Keep Promptfoo for redteam and model scanning.
• Historical eval results are not imported. Archive Promptfoo's output files; anchor future comparisons on a fresh baseline experiment.

FAQ

Can I keep running Promptfoo alongside Langfuse?

Yes, and most teams do during the transition. Promptfoo can fetch prompts from Langfuse via langfuse:// references, so both tools can evaluate the same prompt versions while you port test cases and assertions incrementally. Many teams also keep Promptfoo permanently for red teaming, which Langfuse does not replace.

Do Langfuse evaluators cover Promptfoo's assertion types?

Mostly, but not as a one-to-one catalog. Deterministic assertions (contains, regex, is-json, custom javascript/python) become short evaluator functions or UI-authored code evaluators. Model-graded assertions map to autoevals scorers (factuality, closed QA, similarity, RAG context metrics) or Langfuse-managed LLM-as-a-judge evaluators. Promptfoo's per-assertion weight and threshold scoring has no direct equivalent; compute composite scores in a run-level evaluator instead. Statistical metrics like rouge-n or perplexity need a library call inside an evaluator function.

Can Langfuse fail my CI pipeline the way promptfoo eval does?

Yes. Where promptfoo eval exits with code 100 on failing test cases, a Langfuse experiment script raises RegressionError when a metric drops below your threshold, and the langfuse/experiment-action GitHub Action fails the job and posts a PR comment with scores and a link to the run. Outside GitHub Actions, the same experiment code runs as a pytest or Vitest suite with ordinary assertions.

How do non-engineers access results after migrating?

Through the Langfuse UI, with org- and project-level roles. Experiment runs, score comparisons, and individual traces are browsable without touching the repo, and annotation queues let reviewers grade outputs in a structured workflow. This replaces the Promptfoo pattern of sharing viewer links or result files, and it applies equally on self-hosted Langfuse deployments.

Get help with the migration

Start on Langfuse Cloud or self-host. If you want a migration plan, talk to us.