Docs
Integrations
LiteLLM (Proxy)

πŸš… LiteLLM Integration

LiteLLM (GitHub (opens in a new tab)): Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs).

Integration

By adding the callback you can instantly log your responses across all providers with Langfuse.

main.py
# API keys from project settings in Langfuse
os.environ["LANGFUSE_PUBLIC_KEY"] = "your key"
os.environ["LANGFUSE_SECRET_KEY"] = "your key"
 
# Langfuse host
os.environ["LANGFUSE_HOST"]="https://cloud.langfuse.com" # πŸ‡ͺπŸ‡Ί EU region
# os.environ["LANGFUSE_HOST"]="https://us.cloud.langfuse.com" # πŸ‡ΊπŸ‡Έ US region
 
# Set callback
litellm.success_callback = ["langfuse"]

Full Example

main.py
from litellm import completion
 
## set env variables
os.environ["LANGFUSE_PUBLIC_KEY"] = "your key"
os.environ["LANGFUSE_SECRET_KEY"] = "your key"
 
# Langfuse host
os.environ["LANGFUSE_HOST"]="https://cloud.langfuse.com" # πŸ‡ͺπŸ‡Ί EU region
# os.environ["LANGFUSE_HOST"]="https://us.cloud.langfuse.com" # πŸ‡ΊπŸ‡Έ US region
 
os.environ["OPENAI_API_KEY"] = ""
os.environ["COHERE_API_KEY"] = ""
 
# set callbacks
litellm.success_callback = ["langfuse"]
 
# openai call
response = completion(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Hi πŸ‘‹ - i'm openai"}
  ]
)
 
# cohere call
response = completion(
  model="command-nightly",
  messages=[
    {"role": "user", "content": "Hi πŸ‘‹ - i'm cohere"}
  ]
)

Was this page useful?

Questions? We're here to help

Subscribe to updates