Aug 2, 2023

Langfuse Update — July 2023

Building out the core platform and preparing infrastructure for advanced analytics: Async SDKs, automated token counts, nested Trace UI, langchain integration, public GET API

Hi all, a lot has happened since our launch two weeks ago and I'm excited to share our July update:

The details 👇

🦜🔗 Langchain Integration

For teams building their LLM app with Langchain, adopting Langfuse just got that much easier. We added a CallbackHandler to the Langfuse Python SDK that automatically traces your complex Langchain chains and agents. Simply pass it as a callback.

# Initialize Langfuse handler
from langfuse.callback import CallbackHandler
handler = CallbackHandler(PUBLIC_KEY, SECRET_KEY)
# Setup Langchain
from langchain.chains import LLMChain
chain = LLMChain(llm=llm, prompt=prompt)
# Add Langfuse handler as callback"<user_input", callbacks=[handler])

Integration docs

⚡️ Improved SDKs for Python and JS/TS

Most Langfuse users integrate using our Python or JS/TS SDKs. We made improvements to both DX and performance:

  1. Fully async, avoid adding latency or blocking your app
  2. Typed (Pydantic/Typescript)
  3. Improved DX for nesting traces; no need to manually manage IDs
  4. Support for more runtimes: JS/TS SDKs work in Node.js, Edge (Deno, Vercel, Cloudflare) or in the browser to directly report scores (e.g. user feedback) to Langfuse

SDK docs

⛓️ New Trace UI

Most users of Langfuse run complex LLM-based applications. To help monitor and debug these applications and understand how the different steps lead to the final output, we added a new nested trace UI.

Examples from our Langchain integration:

Debug view in Langfuse

🧮 Automated Token Counts

Usage/cost reporting is a key use case of Langfuse.

Until now, token counts needed to be ingested when logging new LLM calls. For OpenAI and Anthropic models, Langfuse now automatically calculates token counts based on the ingested prompts and completions. This is helpful as some APIs do not provide token counts in their responses. It also reduces integration effort.

ModelTokenizerUsed package
gpt*cl100k_basetiktoken (opens in a new tab)
claude*claude@anthropic-ai/tokenizer (opens in a new tab)

Learn more

🌟 Human-in-the-loop Evaluation

Scores in Langfuse are essential to monitor the quality of your LLM app. Until now, scores were created via the Web SDK based on user feedback (e.g. thumbs up/down, implicit user feedback) or via the API (e.g. when running model-based evals).

Many of you wanted to manually score generations in the UI as you or your team browse production logs. We've added this to the Langfuse UI:

Add manual score in UI

Learn more

👥 User Reporting

Add a userId when ingesting data into Langfuse and use it to get usage/cost broken down by user. Explore what individual users are doing in your LLM app.

More user-based analytics and features are coming soon, e.g. APIs to use this data in your own dashboards and usage-based billing.

User reporting

Learn more

📈 Analytics (alpha)

Currently we are working with a small group of alpha users to test our analytics features. We've focused a lot on getting this right and are excited to share more soon. Reach out if you want to share your needs or want to be part of the alpha. We are onboarding new users every week.

More details

👩‍💻 Public GET API

We added a public GET API to Langfuse to support users consuming Langfuse data in their own applications. Check out the API reference for more details on the available endpoints. Need other endpoints? Shoot me a message: [email protected]

Example use cases:

  • Integrate user feedback (by use case) into your own dashboards
  • Fine-tune on low-quality generations
GET /traces
GET /traces/:traceId
GET /scores
GET /observations/:observationId

API reference

📚 Improved Docs

We've shipped a lot over the last month and the docs were lagging behind. We've now updated the docs to reflect the latest changes and added a bunch of new content. This should make it way easier to get started with Langfuse and incrementally adopt more advanced features. We'd love to get your feedback on the docs, use the new widget on the bottom to 👍/👎, thank you 🙏

We also added a Q&A chatbot as one user was surprised that we don't have one yet. It's still a bit rough around the edges but we'll improve it over time. Give feedback when using it and you can be sure that it is all tracked and analyzed in Langfuse.

New docs

🚢 What's Next?

There is more coming in August. Stay tuned! We'll focus on shipping analytics to all users and further improvements to the UI/DX of the core platform. Anything you'd like to see? Join us on Discord and share your thoughts.

Subscribe to get monthly updates via email:

Follow along on Twitter (@Langfuse (opens in a new tab), @marcklingen (opens in a new tab))

Was this page useful?

Questions? We're here to help