---
title: Langfuse Update — September 2023
date: 2023/10/02
description: "Model-based evals, datasets, core improvements (query engine, complex filters, exports, sharing) and new integrations (Langflow, Flowise, LiteLLM)."
tag: update
ogImage: /images/blog/update-september-2023/og-sep-2023.png
author: Marc
---

Hi everyone 👋, here's a quick overview of all the most notable new features we shipped in September:

- [**Model-based evaluations**](#model-based-eval) via Python SDK

- [**Datasets (beta)**](#datasets) to collect sets of inputs and expected outputs in Langfuse to evaluate your LLM app

- [**In-app analytics**](#analytics) powered by new query engine

- [**New integrations**](#integrations) with Flowise, Langflow and LiteLLM

- **Improvements**
  - [Complex filters](#filters) for all tables
  - [Share traces](#link-sharing) via public link
  - [Export generations](#export-generations) as CSV, JSON or JSONL (e.g. for fine-tuning)

... and many small improvements and bug fixes.

---

The details 👇

## 🚦 Model-based evaluations via Python SDK [#model-based-eval]

We've added an [example implementation](/docs/scores/model-based-evals) on how to run model-based evaluations on production data in Langfuse using the Python SDK.

The new `get_generations` method allows you to fetch all generations based on a filter (e.g. name). You can then run your eval function on each generation and add the scores to Langfuse for exploration.

With this, you can run your favorite eval library (e.g. OpenAI evals, Langkit, Langchain) on all generations in Langfuse.

```python filename="Example (python)"
from langfuse import Langfuse

langfuse = Langfuse(LF_PUBLIC_KEY, LF_SECRET_KEY)
generations = langfuse.get_generations(name="my_generation_name").data

for generation in generations:
    # import function from an eval library, see docs for details
    eval = hallucination_eval(
      generation.prompt,
      generation.completion
    )

    langfuse.score(
      name="hallucination",
      traceId=generation.trace_id,
      observationId=generation.id,
      value=eval["score"],
      comment=eval['reasoning']
    )
```

→ [Docs](/docs/scores/model-based-evals)

## 🗂️ Datasets (beta) [#datasets]

Systematically test new iterations of your LLM app with `datasets`.

Datasets are collections of inputs and expected outputs that you can manage in Langfuse. Upload an existing dataset or create one based on production data (e.g. when discovering new edge cases).

When combined with automated evals, Datasets in Langfuse make it easy to systematically evaluate new iterations of your LLM app.

_Overview of dataset runs on a demo dataset_

![Dataset runs](/images/docs/dataset_runs_table.png)

_Run experiment on dataset_

<LangTabs items={["Python SDK", "JS/TS SDK", "Langchain (Python)"]}>
<Tab>

```python

dataset = langfuse.get_dataset("<dataset_name>")

for item in dataset.items:
    # execute application function and get Langfuse parent observation (span/generation/event, and other observation types: see /docs/observability/features/observation-types)
    # output also returned as it is used to evaluate the run
    generation, output = my_llm_application.run(item.input)

    # link the execution trace to the dataset item and give it a run_name
    item.link(generation, "<run_name>")

    # optionally, evaluate the output to compare different runs more easily
    generation.score(
      name="<example_eval>",
      # any float value
      value=my_eval_fn(
          item.input,
          output,
          item.expected_output
      )
    )
```

</Tab>
<Tab>

```ts
const dataset = await langfuse.getDataset("<dataset_name>");

for (const item of dataset.items) {
  // execute application function and get Langfuse parent observation (span/generation/event, and other observation types: see /docs/observability/features/observation-types)
  // output also returned as it is used to evaluate the run
  const [generation, output] = await myLlmApplication.run(item.input);

  // link the execution trace to the dataset item and give it a run_name
  await item.link(generation, "<run_name>");

  // optionally, evaluate the output to compare different runs more easily
  generation.score({
    name: "<score_name>",
    value: myEvalFunction(item.input, output, item.expectedOutput),
  });
}
```

</Tab>
<Tab>

```python
dataset = langfuse.get_dataset("<dataset_name>")

for item in dataset.items:
    # Langchain callback handler that automatically links the execution trace to the dataset item
    handler = item.get_langchain_handler(run_name="<run_name>")

    # Execute application and pass custom handler
    my_langchain_chain.run(item.input, callbacks=[handler])
```

</Tab>
</LangTabs>

Datasets are currently in beta on Langfuse Cloud as the API might still slightly change. If you'd like to try it, let us know via the in-app chat.

→ [Dataset docs](/docs/datasets/overview)
→ [Python Cookbook](/docs/datasets/python-cookbook)

## 📊 In-app dashboards [#analytics]

Over the last weeks, [analytics features](/docs/analytics/overview) were in public alpha on Langfuse Cloud. We've now shipped a new _query engine_ as an underlying abstraction for the native in-app dashboards. This is a major step towards bringing all analytics features into the Langfuse core project and helps us move much faster on these.

Over the next days, you'll see more and more dashboards popping up in the app. If there is a specific analysis you'd like to see, suggest it on [Discord](/discord).

## 🔄 New integrations [#integrations]

The new integrations make it easier to get started with Langfuse. Thanks to the teams behind Langflow, Flowise and LiteLLM for building/collaborating on these integrations.

See integrations docs for details:

- [Langflow](/docs/langflow): No-code LLM app builder in Python
- [Flowise](/docs/flowise): No-code LLM app builder in JS
- [LiteLLM](/integrations/gateways/litellm): Python library to use any LLM model as drop in replacement of OpenAI API

## 🔎 Complex filters for all tables [#filters]

You can now filter all tables in Langfuse by multiple columns.

## 🌐 Share traces via public link [#link-sharing]

Share traces with anyone via public links. The other person doesn't need a Langfuse account to view the trace.

_Example: https://cloud.langfuse.com/project/clkpwwm0m000gmm094odg11gi/traces/2d6b96f2-0a4d-4366-99a5-1ad558c66e99_

## 💾 Export generations (for fine-tuning) [#export-generations]

In addition to the GET API, you can now directly export generations from the Langfuse UI. Supported formats: CSV, JSON, OpenAI-JSONL.

Use Langfuse to capture high-quality production examples (e.g. from a larger model) and export them for fine-tuning.

  ![Export generations from
  Langfuse](/images/blog/update-september-2023/export_generations.gif)

## 🚢 What's Next?

There is more coming in October. Stay tuned! Based on the new query engine we'll ship extensive dashboards over the next weeks. Anything you'd like to see? Join us on [Discord](/discord) and share your thoughts.

Subscribe to get monthly updates via email:

Subscribe to the Langfuse product update newsletter at https://langfuse.com/changelog.

Follow along on Twitter ([@Langfuse](https://twitter.com/langfuse), [@marcklingen](https://twitter.com/marcklingen))

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/blog/update-2023-09.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.