Self-host an open source AI gateway
Some companies can't send prompts, keys, or usage records through a third-party service, even one that only routes requests. For them, an AI gateway has to run in their own network, next to the rest of their infrastructure. The Langfuse AI Gateway ships with open source Langfuse and deploys with the same Helm chart or Docker Compose setup.
TL;DR: The self-hosted Langfuse AI Gateway is a stateless container that runs next to self-hosted Langfuse. It accepts native OpenAI and Anthropic API requests, replaces gateway keys with your provider connections' API keys, streams responses back, and records every call as a Langfuse generation in your own database. It needs Langfuse v4.75.0 or later on the v4 data model, a DNS name with TLS, and direct outbound HTTPS to your providers or to an internal endpoint. It is in beta, as of November 2026.
![]()
What you get
The self-hosted gateway has the same features as the gateway on Langfuse Cloud:
- Gateway keys instead of provider keys, per developer and per service, with expiry and immediate revocation. See gateway API keys.
- A model allowlist per provider connection, and model patterns per key. See control which AI models your company can use.
- A trace for every call, with model, tokens, cost, latency, errors, and the caller. See AI audit trail.
- Coding agent support for Claude Code, Codex, GitHub Copilot with your own key, and other agents. See coding agent cost tracking.
Self-hosting adds one ingestion mode that Cloud doesn't have: NONE, which routes requests without recording anything, for teams that only want key management.
How it fits into your deployment
| Component | Role | State |
|---|---|---|
| Gateway | Receives client requests, calls the provider, streams the response, uploads the generation | Stateless, no database access |
| Langfuse Web | Resolves gateway keys, stores encrypted provider connections, ingests generations | Uses your Postgres |
| Langfuse Worker, ClickHouse, Redis, S3 | Store and serve traces, like any other Langfuse trace | Your existing stack |
The gateway has no database, Redis, ClickHouse, or S3 credentials. It only talks to Langfuse Web and to the providers. Keep the connection between gateway and web on a private network or encrypted, because web's responses contain provider API keys.
Requirements
- The Helm chart in a release after
2.2.0, or the Docker Compose setup. - Langfuse
v4.75.0or later, with the same version for web and the gateway. With Docker Compose, use a release newer thanv4.75.0. - The v4 data model:
LANGFUSE_MIGRATION_V4_WRITE_MODEmust not belegacy. - A DNS name for the gateway with a TLS certificate, terminated at your Ingress or load balancer.
ENCRYPTION_KEYset on web, because provider connections are stored encrypted with it.
The gateway ignores HTTP_PROXY, HTTPS_PROXY, and other egress proxy
settings, so networks that only allow egress through a forward proxy can't
reach the public provider APIs. To use an internal OpenAI- or
Anthropic-compatible endpoint instead, such as an internal LLM proxy, set a
custom base URL on the
connection and allow its host on both web and the gateway, as described in
Gateway to LLM
providers. See
the self-hosting requirements
for the hosts each provider needs.
Sizing and operations
- Replicas: the gateway is stateless, so scale it horizontally. Run at least two replicas in production and scale on CPU.
- Memory: each instance reserves memory for running requests up to a budget of 1 GiB by default, and returns
503when it is used up. Set the memory limit to at least the budget plus 512 MiB. - Timeouts: requests can stream for up to 600 seconds. Set proxy and load balancer timeouts above 120 seconds,
620to cover the full limit, and turn off response buffering. - Langfuse capacity: every recorded call is ingested like any other trace, so plan Langfuse scaling for gateway traffic as well.
- Health checks: use
/readyfor the load balancer,/healthfor liveness.
Choosing between self-hosted and Cloud
The gateway on Langfuse Cloud needs no deployment and is available in the EU, US, Japan, and HIPAA regions. Self-host it when your data has to stay in your infrastructure, or when you already run Langfuse yourself. For the broader decision, see self-hosted vs Cloud.
FAQ
Is the self-hosted AI gateway open source?
Yes. It is part of open source Langfuse and deploys with the same Helm chart and Docker Compose setup. It is in beta, so its configuration may change in minor releases.
Does the self-hosted gateway work in an air-gapped network?
Yes, if the models are reachable inside the network. Point an OpenAI or Anthropic connection at an internal endpoint with a custom base URL, and allow its host on web and the gateway. The gateway can't reach public provider APIs through an egress proxy.
Can I run the gateway without the rest of Langfuse?
No. The gateway resolves keys and stores provider connections through Langfuse Web, and records generations into Langfuse. With the NONE ingestion mode, it only routes requests and records nothing, but it still needs Langfuse Web.