Integration: 🚅 LiteLLM Proxy
LiteLLM (GitHub): Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs).
This integration is for the LiteLLM Proxy. If you are looking for the LiteLLM SDK integration, see the LiteLLM SDK Integration page.
There are two ways to integrate LiteLLM with Langfuse:
- LiteLLM Proxy with OpenAI SDK Wrapper, the proxy standardizes 100+ models on the OpenAI API schema and the Langfuse OpenAI SDK wrapper instruments the LLM calls.
- LiteLLM Proxy which can send logs to Langfuse if enabled in the UI.
1. LiteLLM Proxy + Langfuse OpenAI SDK Wrapper
This is the preferred way to integrate LiteLLM with Langfuse. The Langfuse OpenAI SDK wrapper automatically captures token counts, latencies, streaming response times (time to first token), API errors, and more.
How this works:
- The LiteLLM Proxy standardizes 100+ models on the OpenAI API schema
- and the Langfuse OpenAI SDK wrapper (Python, JS/TS) instruments the LLM calls.
To see a full end-to-end example, check out the LiteLLM cookbook.
2. Send Logs from LiteLLM Proxy to Langfuse
By setting the callback to Langfuse in the LiteLLM UI you can instantly log your responses across all providers. For more information on how to setup the Proxy UI, see the LiteLLM docs.
You can add additional Langfuse attributes to the requests in order to group requests into traces, add userIds, tags, sessionIds, and more. These attributes are shared across LiteLLM Proxy and SDK, please refer to both documentation pages to learn about all potential options:
Learn more about LiteLLM
What is LiteLLM?
LiteLLM is an open source proxy server to manage auth, loadbalancing, and spend tracking across more than 100 LLMs. LiteLLM has grown to be a popular utility for developers working with LLMs and is universally thought to be a useful abstraction.
Is LiteLLM an Open Source project?
Yes, LiteLLM is open source. The majority of its code is permissively MIT-licensed. You can find the open source LiteLLM repository on GitHub.
Can I use LiteLLM with Ollama and local models?
Yes, you can use LiteLLM with Ollama and other local models. LiteLLM supports all models from Ollama, and it provides a Docker image for an OpenAI API-compatible server for local LLMs like llama2, mistral, and codellama.
How does LiteLLM simplify API calls across multiple LLM providers?
LiteLLM provides a unified interface for calling models such as OpenAI, Anthropic, Cohere, Ollama and others. This means you can call any supported model using a consistent method, such as completion(model, messages), and expect a uniform response format. The library does away with the need for if/else statements or provider-specific code, making it easier to manage and debug LLM interactions in your application.