Dataset Getting Started

This is a step by step guide on how to create your first dataset and run your first experiment. Please refer to the overview for more conceptual introduction.

Creating a dataset

Datasets have a name which is unique within a project.

langfuse.create_dataset(
    name="<dataset_name>",
    # optional description
    description="My first dataset",
    # optional metadata
    metadata={
        "author": "Alice",
        "date": "2022-01-01",
        "type": "benchmark"
    }
)

See low-level SDK docs for details on how to initialize the Python client.

langfuse.createDataset({
  name: "<dataset_name>",
  // optional description
  description: "My first dataset",
  // optional metadata
  metadata: {
    author: "Alice",
    date: "2022-01-01",
    type: "benchmark",
  },
});

Datasets: + New dataset

Create new dataset items

Individual items can be added to a dataset by providing the input and optionally the expected output.

langfuse.create_dataset_item(
    dataset_name="<dataset_name>",
    # any python object or value, optional
    input={
        "text": "hello world"
    },
    # any python object or value, optional
    expected_output={
        "text": "hello world"
    },
    # metadata, optional
    metadata={
        "model": "llama3",
    }
)

See low-level SDK docs for details on how to initialize the Python client.

langfuse.createDatasetItem({
  datasetName: "<dataset_name>",
  // any JS object or value
  input: {
    text: "hello world",
  },
  // any JS object or value, optional
  expectedOutput: {
    text: "hello world",
  },
  // metadata, optional
  metadata: {
    model: "llama3",
  },
});

Datasets > Items: + New item

Create synthetic examples

Frequently, you want to create synthetic examples to test your application to bootstrap your dataset. LLMs are great at generating these by prompting for common questions/tasks.

Once generated, you can upload them to Langfuse via the SDKs.

Create items from production data

In the UI, use + Add to dataset on any observation (span, event, generation) of a production trace.

langfuse.create_dataset_item(
    dataset_name="<dataset_name>",
    input={ "text": "hello world" },
    expected_output={ "text": "hello world" },
    # link to a trace
    source_trace_id="<trace_id>",
    # optional: link to a specific span, event, or generation
    source_observation_id="<observation_id>"
)

langfuse.createDatasetItem({
  datasetName: "<dataset_name>",
  input: { text: "hello world" },
  expectedOutput: { text: "hello world" },
  // link to a trace
  sourceTraceId: "<trace_id>",
  // optional: link to a specific span, event, or generation
  sourceObservationId: "<observation_id>",
});

Edit/archive items

Archiving items will remove them from future experiment runs.

In the UI, you can edit or archive items by clicking on the item in the table.

You can upsert items by providing the id of the item you want to update.

langfuse.create_dataset_item(
    id="<item_id>",
    # example: update status to "ARCHIVED"
    status="ARCHIVED"
)

You can upsert items by providing the id of the item you want to update.

langfuse.createDatasetItem({
  id: "<item_id>",
  // example: update status to "ARCHIVED"
  status: "ARCHIVED",
});

Run experiment on a dataset

When running an experiment on a dataset, the application that shall be tested is executed for each item in the dataset. The execution trace is then linked to the dataset item. This allows to compare different runs of the same application on the same dataset. Each experiment is identified by a run_name.

Optionally, the output of the application can be evaluated to compare different runs more easily. More details on scores/evals here. Options:

Use any evaluation function and directly add a score while running the experiment. See below for implementation details.
Set up model-based evaluation within Langfuse to automatically evaluate the outputs of these runs.

dataset = langfuse.get_dataset("<dataset_name>")
 
for item in dataset.items:
    # Make sure your application function is decorated with @observe decorator to automatically link the trace
    with item.observe(
        run_name="<run_name>",
        run_description="My first run",
        run_metadata={"model": "llama3"},
    ) as trace_id:
        # run your @observe() decorated application on the dataset item input
        output = my_llm_application.run(item.input)
 
        # optionally, evaluate the output to compare different runs more easily
        langfuse.score(
            trace_id=trace_id,
            name="<example_eval>",
            # any float value
            value=my_eval_fn(item.input, output, item.expected_output),
            comment="This is a comment",  # optional, useful to add reasoning
        )
 
# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

See low-level SDK docs for details on how to initialize the Python client and see the Python decorator docs on how to use the @observe decorator for your main application function.

dataset = langfuse.get_dataset("<dataset_name>")
 
for item in dataset.items:
    # execute application function and get langfuse_object (trace/span/generation/event)
    # output also returned as it is used to evaluate the run
    # you can also link using ids, see sdk reference for details
    langfuse_object, output = my_llm_application.run(item.input)
 
    # link the execution trace to the dataset item and give it a run_name
    item.link(
        langfuse_object,
        "<run_name>",
        run_description="My first run", # optional
        run_metadata={ "model": "llama3" } # optional
    )
 
    # optionally, evaluate the output to compare different runs more easily
    langfuse_object.score(
        name="<example_eval>",
        # any float value
        value=my_eval_fn(
            item.input,
            output,
            item.expected_output
        ),
        comment="This is a comment" # optional, useful to add reasoning
    )
 
# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

See low-level SDK docs for details on how to initialize the Python client.

const dataset = await langfuse.getDataset("<dataset_name>");
 
for (const item of dataset.items) {
  // execute application function and get langfuseObject (trace/span/generation/event)
  // output also returned as it is used to evaluate the run
  // you can also link using ids, see sdk reference for details
  const [langfuseObject, output] = await myLlmApplication.run(item.input);
 
  // link the execution trace to the dataset item and give it a run_name
  await item.link(langfuseObject, "<run_name>", {
    description: "My first run", // optional run description
    metadata: { model: "llama3" }, // optional run metadata
  });
 
  // optionally, evaluate the output to compare different runs more easily
  langfuseObject.score({
    name: "<score_name>",
    value: myEvalFunction(item.input, output, item.expectedOutput),
    comment: "This is a comment", // optional, useful to add reasoning
  });
}
 
// Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
await langfuse.flushAsync();

dataset = langfuse.get_dataset("<dataset_name>")
 
for item in dataset.items:
    # Langchain callback handler that automatically links the execution trace to the dataset item
    handler = item.get_langchain_handler(run_name="<run_name>")
 
    # Execute application and pass custom handler
    my_langchain_chain.run(item.input, callbacks=[handler])
 
# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

import { Langfuse } from "langfuse";
import { createDatasetItemHandler } from "langfuse-langchain";
...
 
const langfuse = new Langfuse()
const dataset = await langfuse.getDataset("my-dataset");
const runName = "my-dataset-run";
 
for (const item of dataset.items) {
  // Create langchain handler that automatically links the execution trace to the dataset item run
  const { handler, trace } = await createDatasetItemHandler({ item, runName, langfuseClient: langfuse });
 
  // Pass the callback handler when invoking the chain
  await chain.invoke({ country: item.input }, { callbacks: [handler] });
 
  // Add score to the linked trace depending on the chain output and expected output
  trace.score({
    name: "test-score",
    value: 0.5,
  });
}
 
await langfuse.flushAsync();

dataset = langfuse.get_dataset("<dataset_name>")
 
for item in dataset.items:
    # This context manager will automatically link the execution trace to the dataset item
    with item.observe_llama_index(run_name="<run_name>"):
        # Run your LlamaIndex application on the input
        index = VectorStoreIndex.from_documents([doc1, doc2])
        response = index.as_query_engine().query(item.input)
 
# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

Analyze dataset runs

After each experiment run on a dataset, you can check the aggregated score in the dataset runs table and compare results side-by-side.

Dataset Getting Started

Creating a dataset

Create new dataset items

Run experiment on a dataset

Analyze dataset runs

Was this page useful?

Questions? We're here to help

Subscribe to updates