Core concepts
Beta. The Langfuse AI Gateway is in public beta and can still change.
This page explains how the gateway works: what happens to a request, which keys and settings are involved, and how a model call becomes a trace. To start using it, see Set up the gateway for admins or Send your first request for developers.
Components, architecture and request flow
The gateway serves as a layer between the user's AI requests and the model providers. Users authenticate with the AI gateway and pass on request content. The AI gateway routes the request to the correct model provider, executes the API call, and returns the result back to the user.
On Langfuse Cloud, Langfuse runs the gateway in the EU, US, JP, and HIPAA regions, each with its own gateway host. Gateway traces are stored in your organization's Langfuse region. Requests are forwarded to OpenAI or Anthropic and processed under your agreement with them.
On self-hosted Langfuse, the gateway is a separate service that you deploy next to your instance. See Self-hosting the AI Gateway.
Credential types
The gateway uses three types of keys. Clients hold user keys or service account keys, and only the gateway holds the provider credentials:
| Key type | Origin | Scope | Provided by |
|---|---|---|---|
| Provider credential | Model provider, for example OpenAI or Anthropic | Held only by the gateway. Clients never see it. | Owner/admin |
| User key | AI Gateway | Scoped to one user. Stops working when the user leaves the organization. | User |
| Service account key | AI Gateway | Scoped to a CI or shared service. Not linked to an individual user. | Owner/admin |
Because clients only hold gateway keys, you can rotate a provider key without changing any client, and revoke one person's access without affecting anyone else.
The gateway calls the provider with your own credentials, so the provider bills model usage to your provider account. Langfuse counts each recorded generation as a billable unit.
Model and key routing
An organization can add several provider credentials, including more than one per provider.
The URL path decides which API format the gateway expects. Requests look exactly like requests to the provider. Only the base URL and the key change:
/openai/v1/responsesuses the OpenAI Responses API./openai/v1/chat/completionsuses the OpenAI Chat Completions API./anthropic/v1/messagesuses the Anthropic Messages API.
The gateway doesn't convert one API format into the other, so a client that uses the OpenAI API can only call OpenAI models.
The model ID decides the credential. Each model ID starts with the name of a credential, in the form <credential>:<model>. For example, openai-prod:<model> calls <model> with the credential named openai-prod.
Some coding agents send the provider's model names without a prefix, which also works. See model IDs.
By default, a credential allows every model the provider offers. Admins can turn models off per credential, see Model management.
Tracing
You can enable tracing of the LLM calls in a couple of different tiers (see Tracing for more details):
| Mode | Recorded |
|---|---|
FULL | Everything USAGE records, plus the full input (system prompt, tool definitions, messages, tool calls and results) and the streamed output (text, reasoning or thinking, and tool calls with parsed arguments) |
USAGE (default) | The model, token usage, cost, and latency, without input or output |
NONE (self-hosted only) | Nothing. The gateway only routes requests. |
After the response, the gateway records the call as a generation in Langfuse asynchronously. See how Langfuse captures data for more.
Tracing context around the LLM call
The gateway only sees model calls. It doesn't see what your app does around them, for example retrieval or tool execution. To trace those steps, instrument your app with the Langfuse SDK. To combine both, send the traceparent header of your current span with the request, and the gateway records its generation inside your trace. See trace context and baggage.
Last updated on