TrueFoundry AI Gateway Integration

What is Truefoundry? TrueFoundry is an enterprise-grade AI Gateway and control plane that lets you deploy, govern, and monitor any LLM or Gen-AI workload behind a single OpenAI-compatible API—bringing rate-limiting, cost controls, observability, and on-prem support to production AI applications.

How Truefoundry Integrates with Langfuse

Truefoundry’s AI Gateway and Langfuse combine to give you enterprise-grade observability, governance, and cost control over every LLM request—set up in minutes.

Prerequisites

Before integrating Langfuse with TrueFoundry, ensure you have:

1. TrueFoundry Account: Create a Truefoundry account with atleast one model provider and generate a Personal Access Token by following the instructions in quick start and generating tokens
2. Langfuse Account: Sign up for a free Langfuse Cloud account or self-host Langfuse

%pip install openai langfuse

import os
 
# Langfuse Configuration
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"  # 🇪🇺 EU region
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com"  # 🇺🇸 US region
 
# TrueFoundry Configuration
os.environ["TRUEFOUNDRY_API_KEY"] = "your-truefoundry-token"
os.environ["TRUEFOUNDRY_BASE_URL"] = "https://your-control-plane.truefoundry.cloud/api/llm"

from langfuse import get_client
 
# Test Langfuse authentication
get_client().auth_check()

from langfuse.openai import OpenAI
import os
 
# Initialize OpenAI client with TrueFoundry Gateway
client = OpenAI(
    api_key=os.environ["TRUEFOUNDRY_API_KEY"],
    base_url=os.environ["TRUEFOUNDRY_BASE_URL"]  # Base URL from unified code snippet
)

# Make a request through TrueFoundry Gateway with Langfuse tracing
response = client.chat.completions.create(
    model="openai-main/gpt-4o",  # Paste the model ID you copied from TrueFoundry Gateway
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant specialized in explaining AI concepts."},
        {"role": "user", "content": "Why does an AI gateway help enterprises?"},
    ],
    max_tokens=500,
    temperature=0.7
)
 
print(response.choices[0].message.content)
 
# Ensure all traces are sent to Langfuse
langfuse = get_client()
langfuse.flush()

Advanced Integration with Langfuse Python SDK

Enhance your observability by combining the automatic tracing with additional Langfuse features.

Using the @observe Decorator

The @observe() decorator automatically wraps your functions and adds custom attributes to traces:

from langfuse import observe, get_client
 
langfuse = get_client()
 
@observe()
def analyze_customer_query(query, customer_id):
    """Analyze customer query using TrueFoundry Gateway with full observability"""
    
    response = client.chat.completions.create(
        model="openai-main/gpt-4o",
        messages=[
            {"role": "system", "content": "You are a customer service AI assistant."},
            {"role": "user", "content": query},
        ],
        temperature=0.3
    )
    
    result = response.choices[0].message.content
    
    # Add custom metadata to the trace
    langfuse.update_current_trace(
        input={"query": query, "customer_id": customer_id},
        output={"response": result},
        user_id=customer_id,
        session_id=f"session_{customer_id}",
        tags=["customer-service", "truefoundry-gateway"],
        metadata={
            "model_used": "openai-main/gpt-4o",
            "gateway": "truefoundry",
            "query_type": "customer_support"
        },
        version="1.0.0"
    )
    
    return result
 
# Usage
result = analyze_customer_query("How do I reset my password?", "customer_123")

Debug Mode

Enable debug logging for troubleshooting:

import logging
logging.basicConfig(level=logging.DEBUG)

Note: All other features of Langfuse will work as expected, including prompt management, evaluations, custom dashboards, and advanced observability features. The TrueFoundry integration seamlessly supports the full Langfuse feature set.

Learn More

• TrueFoundry AI Gateway Introduction: https://docs.truefoundry.com/gateway/intro-to-llm-gateway
• TrueFoundry Authentication Guide: https://docs.truefoundry.com/gateway/authentication

Interoperability with the Python SDK

You can use this integration together with the Langfuse Python SDK to add additional attributes to the trace.

from langfuse import observe, get_client
 
langfuse = get_client()
 
@observe()
def my_instrumented_function(input):
    output = my_llm_call(input)
 
    langfuse.update_current_trace(
        input=input,
        output=output,
        user_id="user_123",
        session_id="session_abc",
        tags=["agent", "my-trace"],
        metadata={"email": "[email protected]"},
        version="1.0.0"
    )
 
    return output

from langfuse import get_client
 
langfuse = get_client()
 
with langfuse.start_as_current_span(name="my-trace") as span:
    
    # Run your application here
    output = my_llm_call(input)
 
    # Pass additional attributes to the span
    span.update_trace(
        input=input,
        output=output,
        user_id="user_123",
        session_id="session_abc",
        tags=["agent", "my-trace"],
        metadata={"email": "[email protected]"},
        version="1.0.0"
        )
 
# Flush events in short-lived applications
langfuse.flush()

Next Steps

Once you have instrumented your code, you can manage, evaluate and debug your application: