---
title: "Self-host an open source AI gateway"
description: "Run the open source Langfuse AI Gateway in your own infrastructure: what it does, how it fits next to self-hosted Langfuse, requirements, sizing, and current beta limits."
tags: [guide]
---

# Self-host an open source AI gateway

Some companies can't send prompts, keys, or usage records through a third-party service, even one that only routes requests. For them, an AI gateway has to run in their own network, next to the rest of their infrastructure. The Langfuse AI Gateway ships with open source Langfuse and deploys with the same Helm chart or Docker Compose setup.

**Start with Langfuse** [Sign up](/cloud)

**TL;DR:** The [self-hosted Langfuse AI Gateway](/self-hosting/configuration/ai-gateway) is a stateless container that runs next to self-hosted Langfuse. It accepts native OpenAI and Anthropic API requests, replaces gateway keys with your provider connections' API keys, streams responses back, and records every call as a Langfuse generation in your own database. It needs Langfuse `v4.75.0` or later on the v4 data model, a DNS name with TLS, and direct outbound HTTPS to your providers or to an internal endpoint. It is in beta, as of November 2026.

  ![Langfuse AI Gateway overview with ingestion mode setting and setup steps](/images/docs/ai-gateway/setup-ingestion-mode.png)

## What you get [#what-you-get]

The self-hosted gateway has the same features as the gateway on Langfuse Cloud:

- **Gateway keys instead of provider keys**, per developer and per service, with expiry and immediate revocation. See [gateway API keys](/resources/engineering/ai-gateway-api-keys).
- **A model allowlist** per provider connection, and model patterns per key. See [control which AI models your company can use](/resources/engineering/ai-model-allowlist).
- **A trace for every call**, with model, tokens, cost, latency, errors, and the caller. See [AI audit trail](/resources/engineering/ai-audit-trail).
- **Coding agent support** for Claude Code, Codex, GitHub Copilot with your own key, and other agents. See [coding agent cost tracking](/resources/engineering/coding-agent-cost-tracking).

Self-hosting adds one ingestion mode that Cloud doesn't have: `NONE`, which routes requests without recording anything, for teams that only want key management.

## How it fits into your deployment [#architecture]

| Component                              | Role                                                                                       | State                         |
| -------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------- |
| Gateway                                | Receives client requests, calls the provider, streams the response, uploads the generation | Stateless, no database access |
| Langfuse Web                           | Resolves gateway keys, stores encrypted provider connections, ingests generations          | Uses your Postgres            |
| Langfuse Worker, ClickHouse, Redis, S3 | Store and serve traces, like any other Langfuse trace                                      | Your existing stack           |

The gateway has no database, Redis, ClickHouse, or S3 credentials. It only talks to Langfuse Web and to the providers. Keep the connection between gateway and web on a private network or encrypted, because web's responses contain provider API keys.

## Requirements [#requirements]

- The [Helm chart](/self-hosting/deployment/kubernetes-helm#ai-gateway) in a release after `2.2.0`, or the [Docker Compose](/self-hosting/deployment/docker-compose#ai-gateway) setup.
- Langfuse `v4.75.0` or later, with the same version for web and the gateway. With Docker Compose, use a release newer than `v4.75.0`.
- The v4 data model: `LANGFUSE_MIGRATION_V4_WRITE_MODE` must not be `legacy`.
- A DNS name for the gateway with a TLS certificate, terminated at your Ingress or load balancer.
- `ENCRYPTION_KEY` set on web, because provider connections are stored encrypted with it.

  The gateway ignores `HTTP_PROXY`, `HTTPS_PROXY`, and other egress proxy
  settings, so networks that only allow egress through a forward proxy can't
  reach the public provider APIs. To use an internal OpenAI- or
  Anthropic-compatible endpoint instead, such as an internal LLM proxy, set a
  [custom base URL](/docs/gateway/features/models#custom-base-url) on the
  connection and allow its host on both web and the gateway, as described in
  [Gateway to LLM
  providers](/self-hosting/configuration/ai-gateway#gateway-to-providers). See
  the [self-hosting requirements](/self-hosting/configuration/ai-gateway#requirements)
  for the hosts each provider needs.

## Sizing and operations [#operations]

- **Replicas:** the gateway is stateless, so scale it horizontally. Run at least two replicas in production and scale on CPU.
- **Memory:** each instance reserves memory for running requests up to a budget of 1 GiB by default, and returns `503` when it is used up. Set the memory limit to at least the budget plus 512 MiB.
- **Timeouts:** requests can stream for up to 600 seconds. Set proxy and load balancer timeouts above 120 seconds, `620` to cover the full limit, and turn off response buffering.
- **Langfuse capacity:** every recorded call is ingested like any other trace, so plan [Langfuse scaling](/self-hosting/configuration/scaling) for gateway traffic as well.
- **Health checks:** use `/ready` for the load balancer, `/health` for liveness.

## Choosing between self-hosted and Cloud [#cloud-vs-self-hosted]

The gateway on [Langfuse Cloud](/docs/gateway/overview) needs no deployment and is available in the EU, US, Japan, and HIPAA regions. Self-host it when your data has to stay in your infrastructure, or when you already run Langfuse yourself. For the broader decision, see [self-hosted vs Cloud](/resources/engineering/self-hosted-vs-cloud).

## FAQ

### Is the self-hosted AI gateway open source? [#license]

Yes. It is part of open source Langfuse and deploys with the same Helm chart and Docker Compose setup. It is in beta, so its configuration may change in minor releases.

### Does the self-hosted gateway work in an air-gapped network? [#air-gapped-gateway]

Yes, if the models are reachable inside the network. Point an OpenAI or Anthropic connection at an internal endpoint with a [custom base URL](/docs/gateway/features/models#custom-base-url), and allow its host on web and the gateway. The gateway can't reach public provider APIs through an egress proxy.

### Can I run the gateway without the rest of Langfuse? [#without-langfuse]

No. The gateway resolves keys and stores provider connections through Langfuse Web, and records generations into Langfuse. With the `NONE` ingestion mode, it only routes requests and records nothing, but it still needs Langfuse Web.

[Ask another question](/docs/ask-ai)

**Start with Langfuse** [Sign up](/cloud)

<!-- agent-instructions -->

---

## Agent Instructions

This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/resources/engineering/self-hosted-ai-gateway.md`.

### Querying these docs

If the answer is not on this page, query the documentation instead of guessing:

- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:

  ```bash
  curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
  ```

- **Index of every page**: <https://langfuse.com/llms.txt>, with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).

### Before writing Langfuse code

- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx @langfuse/cli api <resource> <action>`) to read or write traces, prompts, datasets, and scores from the terminal.

Found an error in these docs? Please open an issue at <https://github.com/langfuse/langfuse-docs/issues>.
