Langfuse launches AI Gateway Learn more

Self-host an open source AI gateway

Some companies can't send prompts, keys, or usage records through a third-party service, even one that only routes requests. For them, an AI gateway has to run in their own network, next to the rest of their infrastructure. The Langfuse AI Gateway ships with open source Langfuse and deploys with the same Helm chart or Docker Compose setup.

TL;DR: The self-hosted Langfuse AI Gateway is a stateless container that runs next to self-hosted Langfuse. It accepts native OpenAI and Anthropic API requests, replaces gateway keys with your provider connections' API keys, streams responses back, and records every call as a Langfuse generation in your own database. It needs Langfuse v4.75.0 or later on the v4 data model, a DNS name with TLS, and direct outbound HTTPS to your providers or to an internal endpoint. It is in beta, as of November 2026.

Langfuse AI Gateway overview with ingestion mode setting and setup steps

What you get

The self-hosted gateway has the same features as the gateway on Langfuse Cloud:

Self-hosting adds one ingestion mode that Cloud doesn't have: NONE, which routes requests without recording anything, for teams that only want key management.

How it fits into your deployment

ComponentRoleState
GatewayReceives client requests, calls the provider, streams the response, uploads the generationStateless, no database access
Langfuse WebResolves gateway keys, stores encrypted provider connections, ingests generationsUses your Postgres
Langfuse Worker, ClickHouse, Redis, S3Store and serve traces, like any other Langfuse traceYour existing stack

The gateway has no database, Redis, ClickHouse, or S3 credentials. It only talks to Langfuse Web and to the providers. Keep the connection between gateway and web on a private network or encrypted, because web's responses contain provider API keys.

Requirements

  • The Helm chart in a release after 2.2.0, or the Docker Compose setup.
  • Langfuse v4.75.0 or later, with the same version for web and the gateway. With Docker Compose, use a release newer than v4.75.0.
  • The v4 data model: LANGFUSE_MIGRATION_V4_WRITE_MODE must not be legacy.
  • A DNS name for the gateway with a TLS certificate, terminated at your Ingress or load balancer.
  • ENCRYPTION_KEY set on web, because provider connections are stored encrypted with it.

The gateway ignores HTTP_PROXY, HTTPS_PROXY, and other egress proxy settings, so networks that only allow egress through a forward proxy can't reach the public provider APIs. To use an internal OpenAI- or Anthropic-compatible endpoint instead, such as an internal LLM proxy, set a custom base URL on the connection and allow its host on both web and the gateway, as described in Gateway to LLM providers. See the self-hosting requirements for the hosts each provider needs.

Sizing and operations

  • Replicas: the gateway is stateless, so scale it horizontally. Run at least two replicas in production and scale on CPU.
  • Memory: each instance reserves memory for running requests up to a budget of 1 GiB by default, and returns 503 when it is used up. Set the memory limit to at least the budget plus 512 MiB.
  • Timeouts: requests can stream for up to 600 seconds. Set proxy and load balancer timeouts above 120 seconds, 620 to cover the full limit, and turn off response buffering.
  • Langfuse capacity: every recorded call is ingested like any other trace, so plan Langfuse scaling for gateway traffic as well.
  • Health checks: use /ready for the load balancer, /health for liveness.

Choosing between self-hosted and Cloud

The gateway on Langfuse Cloud needs no deployment and is available in the EU, US, Japan, and HIPAA regions. Self-host it when your data has to stay in your infrastructure, or when you already run Langfuse yourself. For the broader decision, see self-hosted vs Cloud.

FAQ

Is the self-hosted AI gateway open source?

Yes. It is part of open source Langfuse and deploys with the same Helm chart and Docker Compose setup. It is in beta, so its configuration may change in minor releases.

Does the self-hosted gateway work in an air-gapped network?

Yes, if the models are reachable inside the network. Point an OpenAI or Anthropic connection at an internal endpoint with a custom base URL, and allow its host on web and the gateway. The gateway can't reach public provider APIs through an egress proxy.

Can I run the gateway without the rest of Langfuse?

No. The gateway resolves keys and stores provider connections through Langfuse Web, and records generations into Langfuse. With the NONE ingestion mode, it only routes requests and records nothing, but it still needs Langfuse Web.


Was this page helpful?