--- title: AI Gateway (self-hosted) description: Deploy the Langfuse AI Gateway next to your self-hosted Langfuse with the Helm chart, and configure its public URL, TLS, scaling, health checks, and graceful shutdown. label: "Version: v4" sidebarTitle: "AI Gateway" --- # AI Gateway **Beta.** The AI Gateway is in beta on self-hosted deployments. Its configuration, including the `langfuse.gateway.*` Helm values, is not covered by SemVer and may change in minor releases. [Subscribe to Langfuse OSS updates](/self-hosting/oss-newsletter) to hear when it leaves beta. The [Langfuse AI Gateway](/docs/gateway/overview) sends model requests from your applications and coding agents to your LLM providers and records every call as a generation in Langfuse. This guide covers how to deploy and operate the gateway on a self-hosted Langfuse instance. Once it runs, follow [Set up Gateway as an admin](/docs/gateway/get-started/admin-setup) to connect your providers and create API keys. ## Requirements [#requirements] - **Kubernetes with the [Helm chart](/self-hosting/deployment/kubernetes-helm)**, in a release after `2.2.0` that includes the `langfuse.gateway` values. The chart deploys and configures the gateway. The chart is the supported deployment path during the beta: the Docker Compose setup does not include the gateway. To run it another way, run the gateway image with the [gateway variables](#configuration) and set the web variables on Langfuse Web. - **Langfuse `v4.75.0` or later** for web and the gateway image, and the same version for both. Image tags have no `v` prefix, for example `4.75.0`. - **The v4 data model.** The gateway is unavailable while `LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy`, because gateway generations are only stored in the v4 tables. `events_only` (the default) and `dual` work. See the [v4 upgrade guide](/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4#step-2). - **A DNS name for the gateway**, such as `gateway.langfuse.example.com`, with a TLS certificate that covers it. See [TLS](#tls). - **Direct outbound HTTPS from the gateway pods and from web to your LLM providers**, `api.openai.com` and `api.anthropic.com`. Web calls the providers too, to validate credentials and discover models. The gateway ignores `HTTP_PROXY`, `HTTPS_PROXY`, and other egress proxy settings, and it always calls the providers' official API URLs, so Azure OpenAI, Amazon Bedrock, and internal LLM proxies aren't supported. - **`ENCRYPTION_KEY` on web.** Provider credentials are stored encrypted with it, like [LLM connections](/docs/administration/llm-connection). If it changes, stored provider credentials can no longer be read and have to be added again. ## How it works [#architecture] The gateway is a separate, stateless container ([`langfuse/ai-gateway`](https://hub.docker.com/r/langfuse/ai-gateway)) on its own host. It has no database, Redis, ClickHouse, or S3 credentials; it only talks to Langfuse Web and your LLM providers. **Deployment architecture — your infrastructure (VPC / on-premises)** UI, API, and SDK clients send requests to [Langfuse Web](/self-hosting/deployment/infrastructure/containers) (`langfuse/langfuse`), which serves the UI and APIs. [Langfuse Worker](/self-hosting/deployment/infrastructure/containers) (`langfuse/worker`) processes events asynchronously. | Storage service | Purpose | Connections | | --- | --- | --- | | [PostgreSQL](/self-hosting/deployment/infrastructure/postgres) | Transactional data | Web → PostgreSQL; Worker → PostgreSQL | | [Redis / Valkey](/self-hosting/deployment/infrastructure/cache) | Cache and job queue | Web → Redis / Valkey; Redis / Valkey → Worker | | [ClickHouse](/self-hosting/deployment/infrastructure/clickhouse) | Observability data | Web → ClickHouse; Worker → ClickHouse | | [S3 / Blob storage](/self-hosting/deployment/infrastructure/blobstorage) | Raw events and multi-modal attachments | Web → S3 / Blob storage; Worker → S3 / Blob storage | Both containers can optionally connect to [LLM providers](/self-hosting/deployment/infrastructure/llm-api): Web for the playground, Worker for evaluations. Bring your own LLM API; it can also run in the same VPC or a peered VPC. The optional [Langfuse AI Gateway](/self-hosting/configuration/ai-gateway) (`langfuse/ai-gateway`, beta) receives model requests from apps and coding agents. Gateway → Web resolves gateway API keys and uploads traces; Gateway → LLM providers relays the requests. 1. A client sends a request to the gateway's public URL, for example `https://gateway.langfuse.example.com/anthropic/v1/messages`, with a gateway API key. 2. The gateway asks Langfuse Web to resolve the key. Web checks the key and returns the organization's provider credential. 3. The gateway forwards the native request to the provider, with the provider credential in place of the gateway key, and streams the response back. See [what the gateway changes](/faq/all/ai-gateway-request-changes). 4. The gateway uploads the generation to Langfuse Web, which ingests it like any other trace. Organization admins then add provider credentials and gateway API keys in the Langfuse UI. For how clients, keys, and providers fit together, see [Concepts](/docs/gateway/concepts). ## Deploy with Helm [#helm] Enable the gateway and give it its own Ingress host in your `values.yaml`: ```yaml filename="values.yaml" langfuse: nextauth: url: https://langfuse.example.com gateway: enabled: true ingress: enabled: true className: nginx hosts: - gateway.langfuse.example.com tls: enabled: true secretName: langfuse-gateway-tls annotations: # Optional: let cert-manager issue the certificate, see TLS below. cert-manager.io/cluster-issuer: letsencrypt # Streams can run for up to 600 s and stay quiet for up to 120 s. nginx.ingress.kubernetes.io/proxy-read-timeout: "620" nginx.ingress.kubernetes.io/proxy-send-timeout: "620" nginx.ingress.kubernetes.io/proxy-buffering: "off" nginx.ingress.kubernetes.io/proxy-request-buffering: "off" # The gateway accepts request bodies of up to 16 MiB. nginx.ingress.kubernetes.io/proxy-body-size: "17m" resources: requests: cpu: "1" memory: 2Gi limits: memory: 2Gi ``` The chart pulls the gateway image [`langfuse/ai-gateway`](https://hub.docker.com/r/langfuse/ai-gateway) from Docker Hub through `docker.langfuse.com`, like the [web and worker images](/self-hosting/deployment/infrastructure/containers#container-images). The image tag follows `langfuse.image.tag` and the chart's `appVersion`, like web. If your chart's `appVersion` is older than `4.75.0`, set `langfuse.image.tag` to a newer release. Then apply it with `helm upgrade` and check that the gateway pods become ready: ```bash helm upgrade langfuse langfuse/langfuse -n langfuse -f values.yaml kubectl get pods -n langfuse -l app.kubernetes.io/component=gateway ``` With `langfuse.gateway.enabled: true`, the chart: - Deploys the gateway as its own Deployment, Service (port `8080`), Ingress, optional HorizontalPodAutoscaler, and PodDisruptionBudget. - Sets `LANGFUSE_AI_GATEWAY_DEPLOYED=true` on web, which turns on the gateway pages in the UI for all organizations. - Points the gateway at web's in-cluster Service, so gateway traffic to web does not leave the cluster. See [Gateway to Langfuse Web](#gateway-to-web). - Generates the two shared secrets into the `-app` Secret: the service key that web and the gateway use to authenticate each other, and the private key web uses to sign the gateway's trace upload tokens. With `helm template`-based GitOps tools such as Argo CD, the chart cannot read existing Secrets, so the generated keys change on every sync and gateway requests fail until both web and gateway restart. In that case, create the keys yourself and reference them with `langfuse.gateway.serviceKey.secretKeyRef` and `langfuse.gateway.jwt.privateKey.secretKeyRef`: ```bash kubectl create secret generic langfuse-gateway -n langfuse \ --from-literal=ai-gateway-service-key="$(openssl rand -hex 32)" \ --from-file=ai-gateway-jwt-private-key=<(openssl ecparam -name prime256v1 -genkey -noout) ``` ```yaml filename="values.yaml" langfuse: gateway: serviceKey: secretKeyRef: name: langfuse-gateway key: ai-gateway-service-key jwt: privateKey: secretKeyRef: name: langfuse-gateway key: ai-gateway-jwt-private-key ``` For the full list of values, see the [chart README](/self-hosting/deployment/kubernetes-helm#readme). ### Public URL [#public-url] Langfuse shows the gateway's public URL in the UI and in the setup snippets for coding agents. The chart sets it on web as `LANGFUSE_AI_GATEWAY_PUBLIC_URL`, in this order: 1. `langfuse.gateway.publicUrl`, if set. 2. Otherwise the first host in `langfuse.gateway.ingress.hosts`, over `https://` when the gateway Ingress has TLS enabled or `langfuse.nextauth.url` starts with `https://`, and `http://` otherwise. 3. Without either, web shows `gateway.`. Set `langfuse.gateway.publicUrl` explicitly when you expose the gateway in another way, such as your own Ingress, the Gateway API, or a `LoadBalancer` Service. The value must be an origin without a path, such as `https://llm.example.com`. ### TLS [#tls] We strongly recommend serving the gateway over HTTPS: - Every request carries a gateway API key, and request and response bodies contain full prompts and model outputs. - Coding clients such as Claude Desktop expect an `https://` gateway URL. Like Langfuse Web, the gateway does not terminate TLS itself. Terminate TLS in front of it, at the Ingress controller or at a load balancer. The gateway's host needs its own DNS record and a certificate that covers it. The chart doesn't need cert-manager for the gateway: use it only if you already issue certificates with it, otherwise bring your own certificate or terminate TLS at a load balancer. **cert-manager** If cert-manager runs in your cluster, add its issuer annotation to the gateway Ingress. cert-manager then issues and renews a certificate for the gateway host and stores it in `langfuse.gateway.ingress.tls.secretName`: ```yaml filename="values.yaml" langfuse: gateway: ingress: enabled: true hosts: - gateway.langfuse.example.com tls: enabled: true secretName: langfuse-gateway-tls annotations: cert-manager.io/cluster-issuer: letsencrypt ``` Use a `langfuse.gateway.ingress.tls.secretName` that differs from web's `langfuse.ingress.tls.secretName`, so the two certificates do not overwrite each other. With an HTTP-01 issuer, the gateway host must be reachable from the internet; for internal hosts, use a DNS-01 issuer. **Your own certificate** Create a Kubernetes TLS Secret from your certificate and key, and reference it without a cert-manager annotation: ```bash kubectl create secret tls langfuse-gateway-tls -n langfuse \ --cert=gateway.crt --key=gateway.key ``` ```yaml filename="values.yaml" langfuse: gateway: ingress: enabled: true hosts: - gateway.langfuse.example.com tls: enabled: true secretName: langfuse-gateway-tls ``` If web's certificate already covers the gateway host, as a wildcard such as `*.langfuse.example.com` or as a subject alternative name, you can reference web's Secret instead. Include the full chain in `gateway.crt`, since clients reject certificates whose intermediates are missing. **TLS at a load balancer** If TLS terminates at a cloud load balancer, for example an AWS Application Load Balancer, leave `langfuse.gateway.ingress.tls.enabled` off and configure the certificate on the load balancer. The chart still uses `https://` for the [public URL](#public-url) when `langfuse.nextauth.url` is `https://`; otherwise set `langfuse.gateway.publicUrl`. With the AWS Load Balancer Controller: ```yaml filename="values.yaml" langfuse: gateway: ingress: enabled: true className: alb hosts: - gateway.langfuse.example.com annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:::certificate/ alb.ingress.kubernetes.io/healthcheck-path: /ready alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=620 ``` The idle timeout applies to the whole load balancer. If the gateway shares web's load balancer through `alb.ingress.kubernetes.io/group.name`, the timeout changes for web too, so use a separate group for the gateway. To verify, check that the gateway answers over HTTPS with a valid certificate: ```bash curl -v https://gateway.langfuse.example.com/health # {"status":"ok"} ``` ### Gateway to Langfuse Web [#gateway-to-web] The gateway calls Langfuse Web to resolve every request and to upload generations. Web's responses contain the provider API keys of your organization, so this connection must stay private or be encrypted. By default, the chart points the gateway at web's in-cluster Service, `http://-web..svc.cluster.local:3000`. This traffic stays inside your cluster network, avoids a round trip through the Ingress, and keeps request resolution fast. Because the URL is plain HTTP, the chart sets `LANGFUSE_AI_GATEWAY_ALLOW_HTTP_WEB_URL=true` on the gateway. Without that setting, the gateway only accepts `https://` web URLs. For end-to-end encryption inside the cluster, use a service mesh with mutual TLS, as described in [Encryption](/self-hosting/configuration/encryption#https). If the gateway cannot reach web's Service, for example because web runs in another cluster, set `langfuse.gateway.webUrl` to an `https://` URL such as your `NEXTAUTH_URL`. Requests then pass through web's Ingress, so its body size limit must allow the gateway's requests. The chart's in-cluster default has no path, so if web runs under a [custom base path](/self-hosting/configuration/custom-base-path), set `langfuse.gateway.webUrl` including the path, for example `http://langfuse-web.langfuse.svc.cluster.local:3000/langfuse`. If web's certificate is signed by a private CA, mount the CA bundle with `langfuse.gateway.pod.extraVolumes` and `langfuse.gateway.pod.extraVolumeMounts`, and point `SSL_CERT_FILE` at it with `langfuse.gateway.pod.additionalEnv`. `SSL_CERT_FILE` replaces the gateway's trust store for all connections, including the ones to your LLM providers. The bundle must contain the public root certificates as well as your private CA, or provider calls fail with TLS errors. Never use a plain `http://` web URL across a public network. Web's responses contain provider API keys, and the gateway's requests carry the service key and upload tokens. ### Proxy timeouts and buffering [#timeouts] Model responses are streamed and can take minutes. The gateway ends a request after 600 seconds (`LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS`) or after 120 seconds without progress (`LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS`). Everything between the client and the gateway must allow this: - Set proxy read and send timeouts, or the load balancer idle timeout, above 120 seconds. `620` covers the full 600 seconds. - Turn off response buffering, so streamed tokens reach the client as they arrive. - Allow request bodies of at least 16 MiB (`LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES`). The chart sets no proxy annotations by default, because they depend on your Ingress controller. See the examples in [Deploy with Helm](#helm) for ingress-nginx and in [TLS](#tls) for the AWS Load Balancer Controller. ## Operations [#operations] ### Scaling [#scaling] The gateway is stateless, so scale it horizontally by adding pods. On Langfuse Cloud, we run several gateway instances per region and scale them on CPU utilization. - **Run at least two replicas in production**, so that restarting a pod or a node doesn't interrupt all traffic. The chart keeps all but one pod running during node maintenance. - **Enable the HorizontalPodAutoscaler** with `langfuse.gateway.hpa.enabled: true` and scale on CPU. The chart defaults (`langfuse.gateway.hpa.minReplicas: 1`, `langfuse.gateway.hpa.maxReplicas: 2`) are a starting point; raise them for production traffic. - **Size memory with the memory budget.** Each pod reserves memory for every running request from `LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES` (1 GiB by default), and returns `503` right away when the budget is used up. Set the pod's memory limit to at least the budget plus 512 MiB. To serve more concurrent or larger requests per pod, raise the budget and the limit together, or add pods. - **Remove replicas slowly when scaling down**, so long-running streaming requests aren't cut off. By default, Kubernetes waits 5 minutes of lower load before it removes replicas; don't shorten that. Requests still running on a stopping pod get up to 5 minutes to finish, see [Graceful shutdown](#graceful-shutdown). - **Scale Langfuse too.** Every request makes web resolve the gateway key (cached for 60 seconds), and every generation is ingested like other traces. Plan web and worker capacity for the added trace volume, see [Scaling](/self-hosting/configuration/scaling). To watch the gateway, set `OTEL_EXPORTER_OTLP_ENDPOINT` on it. It then exports operational traces and metrics, including `gateway.memory.reserved.peak` and `gateway.admission.rejected`, to your OpenTelemetry collector. ### Health checks [#health-checks] The gateway serves two unauthenticated endpoints on port `8080`, which the chart uses as Kubernetes probes: | Endpoint | Probe | Response | | --------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/health` | Liveness | `200 {"status":"ok"}` while the process serves HTTP, including during shutdown. It does not check dependencies. | | `/ready` | Readiness | `200 {"status":"ready"}` once the gateway is configured with a web URL and service key. `503 {"status":"unconfigured"}` without that configuration and `503 {"status":"draining"}` during shutdown. | `/ready` does not call web or the providers, so it stays `200` when they are unreachable. Those failures show up as errors on individual requests and in the gateway logs. Use `/ready` for load balancer health checks. ### Graceful shutdown [#graceful-shutdown] Deployments and scaling down stop gateway pods while clients may still be streaming. The chart configures the pod so that running streams can finish: 1. **`langfuse.gateway.preStopSleepSeconds` (10 s):** the pod keeps serving while Kubernetes removes it from the Service and Ingress, so no new requests arrive. 2. **`langfuse.gateway.connectionDrainTimeoutSeconds` (300 s):** after `SIGTERM`, the gateway rejects new requests, reports `503` on `/ready`, and lets requests that are still running finish. 3. **`langfuse.gateway.telemetryShutdownTimeoutSeconds` (30 s):** after that, the gateway uploads the remaining generations to Langfuse. 4. **`langfuse.gateway.terminationGracePeriodSeconds` (350 s):** Kubernetes kills the pod after this time. It must be greater than the sum of the three steps above; the chart rejects values that are not. Streams that run longer than `langfuse.gateway.connectionDrainTimeoutSeconds` are cut on every deployment and when scaling down. This timeout is capped at 300 seconds, while requests can run for up to 600 seconds, so the longest streams can still be cut. ### Rotating keys [#rotate-keys] Web accepts a previous service key and a previous JWT public key next to the current ones. Rotate in three deployments so that every web pod accepts both keys while pods restart. Set the `_PREVIOUS` variables on web with `langfuse.web.pod.additionalEnv`, and pin the current keys with `secretKeyRef` as shown in [Deploy with Helm](#helm). **Service key**, shared by web and the gateway: 1. On web, set the **new** key as `LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS` and redeploy web. Web now accepts both keys. 2. Switch `langfuse.gateway.serviceKey` to the new key, which changes it on web and the gateway, and in the same deployment set `LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS` to the **old** key. 3. Remove `LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS` and redeploy web. **JWT signing key**, used only by web: 1. On web, set the **new** public key as `LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_PUBLIC_KEY` and redeploy web. Derive a public key with `openssl ec -in private-key.pem -pubout`. 2. Switch `langfuse.gateway.jwt.privateKey` to the new private key, and in the same deployment set `LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_PUBLIC_KEY` to the **old** public key. 3. Wait at least 15 minutes, the lifetime of upload tokens signed with the old key. Then remove the previous public key and redeploy web. Key IDs are derived from the public keys. Set `LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_KEY_ID` only if you set `LANGFUSE_AI_GATEWAY_JWT_KEY_ID` explicitly. ## Configuration reference [#configuration] The chart sets these variables for you. You only need them when you deploy the gateway without the chart or override the chart's values. **Gateway container** | Variable | Default | Description | | -------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `LANGFUSE_AI_GATEWAY_WEB_URL` | Required | Base URL of Langfuse Web, including any [custom base path](/self-hosting/configuration/custom-base-path). Without it, `/ready` returns `503` and inference is disabled. With the chart, set it with `langfuse.gateway.webUrl`. | | `LANGFUSE_AI_GATEWAY_SERVICE_KEY` | Required | Shared key that authenticates the gateway to web. Must match the value on web. | | `LANGFUSE_AI_GATEWAY_ALLOW_HTTP_WEB_URL` | `false` | Set to `true` to accept a plain `http://` web URL. Only for private networks such as web's in-cluster Service. | | `LANGFUSE_AI_GATEWAY_LISTEN_ADDRESS` | `0.0.0.0:8080` | Address and port the gateway listens on. | | `LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS` | `10` | Seconds to drain running requests after `SIGTERM`, from 1 to 300. The chart sets `300`. | | `LANGFUSE_AI_GATEWAY_TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Seconds to upload remaining generations after the drain, from 1 to 300. The chart sets `30`. | | `LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES` | `16777216` | Largest accepted request body (16 MiB). | | `LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES` | `1073741824` | Memory that all running requests of one pod may reserve together (1 GiB). Set the pod's memory limit to at least this value plus 512 MiB. | | `LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS` | `600` | Longest time a request may run, at least 60. Upload tokens last 15 minutes, so generations of longer requests may be dropped. | | `LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS` | `120` | Longest time a request may go without progress. | | `LANGFUSE_AI_GATEWAY_REQUEST_MEMORY_OVERHEAD_BYTES` | `2097152` | Memory every running request reserves on top of its body (2 MiB), for response capture and buffers. | | `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` | Memory for generations waiting to be uploaded (64 MiB), from 4 MiB to 4 GiB. | | `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | | Deprecated name of `LANGFUSE_AI_GATEWAY_CONNECTION_DRAIN_TIMEOUT_SECONDS`. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | | Optional OTLP/HTTP endpoint for the gateway's own operational traces and metrics. Unrelated to the generations it records in Langfuse. | | `OTEL_SERVICE_NAME` | `ai-gateway` | Service name of the operational traces. | | `OTEL_TRACES_SAMPLER_ARG` | | Sampling ratio of the operational traces, from 0 to 1. | | `LANGFUSE_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`, or `fatal`. The gateway does not accept `trace`; the chart maps it to `debug`. | | `LANGFUSE_LOG_FORMAT` | `text` | `text` or `json`. | Set additional gateway variables with `langfuse.gateway.pod.additionalEnv`. The global `langfuse.additionalEnv` does not apply to the gateway. **Web container** | Variable | Default | Description | | ----------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `LANGFUSE_AI_GATEWAY_DEPLOYED` | `false` | Set to `true` when a gateway runs next to web. While `false`, the UI shows that the gateway is not deployed. | | `LANGFUSE_AI_GATEWAY_PUBLIC_URL` | Derived | Gateway URL shown in the UI and setup snippets. See [Public URL](#public-url). | | `LANGFUSE_AI_GATEWAY_SERVICE_KEY` | Required | Same value as on the gateway. | | `LANGFUSE_AI_GATEWAY_JWT_PRIVATE_KEY` | Required | ES256 (P-256) private key in PEM format that signs the gateway's upload tokens. Generate it with `openssl ecparam -name prime256v1 -genkey -noout`. | | `LANGFUSE_AI_GATEWAY_JWT_PUBLIC_KEY` | Derived | Optional. Derived from the private key when unset. | | `LANGFUSE_AI_GATEWAY_JWT_KEY_ID` | Derived | Optional. Derived from the public key when unset. | | `LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS` | | Previous service key, accepted during [rotation](#rotate-keys). | | `LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_PUBLIC_KEY` | | Previous JWT public key, accepted during [rotation](#rotate-keys). | | `LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_KEY_ID` | Derived | Optional. Key ID of the previous public key. | | `LANGFUSE_AI_GATEWAY_CACHE_RESOLVE_ENABLED` | `true` | Caches key lookups for the gateway. | | `LANGFUSE_AI_GATEWAY_CACHE_RESOLVE_TTL_SECONDS` | `60` | How long a cached key lookup is used. Bounds how long a role change that doesn't delete a key takes to apply. | | `ENCRYPTION_KEY` | Required | Encrypts stored provider credentials. Changing it makes them unreadable. | ## Troubleshooting [#troubleshooting] ### The UI says the AI Gateway isn't deployed [#ui-not-deployed] Web does not have `LANGFUSE_AI_GATEWAY_DEPLOYED=true`. Check that `langfuse.gateway.enabled` is `true` and that web runs `v4.75.0` or later. If the gateway pages are still missing, check that `LANGFUSE_MIGRATION_V4_WRITE_MODE` is not `legacy`. ### Gateway pods are not ready [#pods-not-ready] `/ready` returns `503` until the gateway has a web URL and service key. Check the pod logs: invalid configuration fails at startup with the name of the offending variable. ### The gateway image can't be pulled [#image-pull-fails] The chart pulls `docker.langfuse.com/langfuse/ai-gateway`. If your cluster can't reach `docker.langfuse.com`, set `langfuse.gateway.image.repository` to `langfuse/ai-gateway` to pull from Docker Hub directly, or mirror the image to your own registry. ### Requests fail with `401` [#requests-fail-401] Check that the client sends a gateway API key, not a Langfuse project API key, and that the key has not been revoked. For client-side causes, see [AI Gateway errors](/faq/all/ai-gateway-error-codes). ### Every request fails while `/ready` returns `200` [#requests-fail-ready-200] `/ready` does not check web. Check the gateway logs for failed calls to web: either the gateway cannot reach `LANGFUSE_AI_GATEWAY_WEB_URL`, or the service key differs between web and the gateway. To change the service key without failing requests, follow [Rotating keys](#rotate-keys). ### Requests fail with `503` under load [#requests-fail-503] The pod's [memory budget](#scaling) is used up by running requests. Add replicas, or raise `LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES` together with the pod's memory limit. ### Long responses stop after 60 seconds or arrive all at once [#streams-stop] A proxy in front of the gateway has a short timeout or buffers responses. See [Proxy timeouts and buffering](#timeouts). ### Large requests fail with `413` [#requests-fail-413] Raise the body size limit of your Ingress controller, for example `nginx.ingress.kubernetes.io/proxy-body-size: "17m"`. ### Streams break during deployments [#streams-break-deploy] Streams longer than `langfuse.gateway.connectionDrainTimeoutSeconds` are cut when a pod stops. Check the values in [Graceful shutdown](#graceful-shutdown) and that `langfuse.gateway.terminationGracePeriodSeconds` is greater than their sum. ### Requests succeed but no generations appear in Langfuse [#no-generations] The gateway uploads generations to web after each request. Check the gateway logs for upload errors, and check that web runs the v4 data model, not the `legacy` write mode. ### Provider calls fail with TLS errors after you set `SSL_CERT_FILE` [#tls-errors-ssl-cert-file] The bundle replaces the trust store, so it must also contain the public root certificates. See [Gateway to Langfuse Web](#gateway-to-web). ### Clients report certificate errors [#certificate-errors] The certificate must cover the gateway host and include the full chain. Test with `curl -v https:///health`. [Ask another question](/docs/ask-ai) When you report a problem, include the `langfuse-request-id` response header of the failing request. For problems with the gateway in your organization's setup, see the [AI Gateway troubleshooting](/faq/all/ai-gateway-error-codes). --- If you experience any issues when self-hosting Langfuse, please: 1. Check out [Troubleshooting & FAQ](/self-hosting/troubleshooting-and-faq) page. 2. Use [Ask AI](/docs/ask-ai) to get instant answers to your questions. 3. Ask the maintainers on [GitHub Discussions](/gh-support). 4. Create a bug report or feature request on [GitHub](/issues). Enterprise-grade support is available when self-hosting Langfuse. Learn more on our [pricing page](/pricing-self-host). --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/self-hosting/configuration/ai-gateway.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx @langfuse/cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .