Langfuse Update — July 2023
Building out the core platform and preparing infrastructure for advanced analytics: Async SDKs, automated token counts, nested Trace UI, langchain integration, public GET API
Hi all, a lot has happened since our launch two weeks ago and I'm excited to share our July update:
- Faster integration
- New nested Trace UI
- More metrics
The details 👇
For teams building their LLM app with Langchain, adopting Langfuse just got that much easier. We added a
CallbackHandler to the Langfuse Python SDK that automatically traces your complex Langchain chains and agents. Simply pass it as a callback.
# Initialize Langfuse handler from langfuse.callback import CallbackHandler handler = CallbackHandler(PUBLIC_KEY, SECRET_KEY) # Setup Langchain from langchain.chains import LLMChain ... chain = LLMChain(llm=llm, prompt=prompt) # Add Langfuse handler as callback chain.run(input="<user_input", callbacks=[handler])
Most Langfuse users integrate using our Python or JS/TS SDKs. We made improvements to both DX and performance:
- Fully async, avoid adding latency or blocking your app
- Typed (Pydantic/Typescript)
- Improved DX for nesting traces; no need to manually manage IDs
- Support for more runtimes: JS/TS SDKs work in Node.js, Edge (Deno, Vercel, Cloudflare) or in the browser to directly report scores (e.g. user feedback) to Langfuse
→ SDK docs
Most users of Langfuse run complex LLM-based applications. To help monitor and debug these applications and understand how the different steps lead to the final output, we added a new nested trace UI.
Examples from our Langchain integration:
Usage/cost reporting is a key use case of Langfuse.
Until now, token counts needed to be ingested when logging new LLM calls. For OpenAI and Anthropic models, Langfuse now automatically calculates token counts based on the ingested prompts and completions. This is helpful as some APIs do not provide token counts in their responses. It also reduces integration effort.
Scores in Langfuse are essential to monitor the quality of your LLM app. Until now, scores were created via the Web SDK based on user feedback (e.g. thumbs up/down, implicit user feedback) or via the API (e.g. when running model-based evals).
Many of you wanted to manually score generations in the UI as you or your team browse production logs. We've added this to the Langfuse UI:
userId when ingesting data into Langfuse and use it to get usage/cost broken down by user. Explore what individual users are doing in your LLM app.
More user-based analytics and features are coming soon, e.g. APIs to use this data in your own dashboards and usage-based billing.
Currently we are working with a small group of alpha users to test our analytics features. We've focused a lot on getting this right and are excited to share more soon. Reach out if you want to share your needs or want to be part of the alpha. We are onboarding new users every week.
We added a public GET API to Langfuse to support users consuming Langfuse data in their own applications. Check out the API reference for more details on the available endpoints. Need other endpoints? Shoot me a message: [email protected]
Example use cases:
- Integrate user feedback (by use case) into your own dashboards
- Fine-tune on low-quality generations
GET /traces GET /traces/:traceId GET /scores GET /observations/:observationId
We've shipped a lot over the last month and the docs were lagging behind. We've now updated the docs to reflect the latest changes and added a bunch of new content. This should make it way easier to get started with Langfuse and incrementally adopt more advanced features. We'd love to get your feedback on the docs, use the new widget on the bottom to 👍/👎, thank you 🙏
We also added a Q&A chatbot as one user was surprised that we don't have one yet. It's still a bit rough around the edges but we'll improve it over time. Give feedback when using it and you can be sure that it is all tracked and analyzed in Langfuse.
There is more coming in August. Stay tuned! We'll focus on shipping analytics to all users and further improvements to the UI/DX of the core platform. Anything you'd like to see? Join us on Discord and share your thoughts.
Subscribe to get monthly updates via email: