AI Gateway

The Langfuse AI Gateway sends model requests from your applications and coding agents to your LLM providers and records every call as a generation in Langfuse. This guide covers how to deploy and operate the gateway on a self-hosted Langfuse instance. Once it runs, follow Set up Gateway as an admin to connect your providers and create API keys.

Requirements

• Kubernetes with the Helm chart, in a release after 2.2.0 that includes the langfuse.gateway values. The chart deploys and configures the gateway. The chart is the supported deployment path during the beta: the Docker Compose setup does not include the gateway. To run it another way, run the gateway image with the gateway variables and set the web variables on Langfuse Web.
• Langfuse v4.75.0 or later for web and the gateway image, and the same version for both. Image tags have no v prefix, for example 4.75.0.
• The v4 data model. The gateway is unavailable while LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, because gateway generations are only stored in the v4 tables. events_only (the default) and dual work. See the v4 upgrade guide.
• A DNS name for the gateway, such as gateway.langfuse.example.com, with a TLS certificate that covers it. See TLS.
• Direct outbound HTTPS from the gateway pods and from web to your LLM providers, api.openai.com and api.anthropic.com. Web calls the providers too, to validate credentials and discover models. The gateway ignores HTTP_PROXY, HTTPS_PROXY, and other egress proxy settings, and it always calls the providers' official API URLs, so Azure OpenAI, Amazon Bedrock, and internal LLM proxies aren't supported.
• ENCRYPTION_KEY on web. Provider credentials are stored encrypted with it, like LLM connections. If it changes, stored provider credentials can no longer be read and have to be added again.

How it works

The gateway is a separate, stateless container (langfuse/ai-gateway) on its own host. It has no database, Redis, ClickHouse, or S3 credentials; it only talks to Langfuse Web and your LLM providers.

1. A client sends a request to the gateway's public URL, for example https://gateway.langfuse.example.com/anthropic/v1/messages, with a gateway API key.
2. The gateway asks Langfuse Web to resolve the key. Web checks the key and returns the organization's provider credential.
3. The gateway forwards the native request to the provider, with the provider credential in place of the gateway key, and streams the response back. See what the gateway changes.
4. The gateway uploads the generation to Langfuse Web, which ingests it like any other trace.

Organization admins then add provider credentials and gateway API keys in the Langfuse UI. For how clients, keys, and providers fit together, see Concepts.

Deploy with Helm

Enable the gateway and give it its own Ingress host in your values.yaml:

langfuse:
  nextauth:
    url: https://langfuse.example.com
  gateway:
    enabled: true
    ingress:
      enabled: true
      className: nginx
      hosts:
        - gateway.langfuse.example.com
      tls:
        enabled: true
        secretName: langfuse-gateway-tls
      annotations:
        # Optional: let cert-manager issue the certificate, see TLS below.
        cert-manager.io/cluster-issuer: letsencrypt
        # Streams can run for up to 600 s and stay quiet for up to 120 s.
        nginx.ingress.kubernetes.io/proxy-read-timeout: "620"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "620"
        nginx.ingress.kubernetes.io/proxy-buffering: "off"
        nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
        # The gateway accepts request bodies of up to 16 MiB.
        nginx.ingress.kubernetes.io/proxy-body-size: "17m"
    resources:
      requests:
        cpu: "1"
        memory: 2Gi
      limits:
        memory: 2Gi

The chart pulls the gateway image langfuse/ai-gateway from Docker Hub through docker.langfuse.com, like the web and worker images. The image tag follows langfuse.image.tag and the chart's appVersion, like web. If your chart's appVersion is older than 4.75.0, set langfuse.image.tag to a newer release.

Then apply it with helm upgrade and check that the gateway pods become ready:

helm upgrade langfuse langfuse/langfuse -n langfuse -f values.yaml
kubectl get pods -n langfuse -l app.kubernetes.io/component=gateway

With langfuse.gateway.enabled: true, the chart:

• Deploys the gateway as its own Deployment, Service (port 8080), Ingress, optional HorizontalPodAutoscaler, and PodDisruptionBudget.
• Sets LANGFUSE_AI_GATEWAY_DEPLOYED=true on web, which turns on the gateway pages in the UI for all organizations.
• Points the gateway at web's in-cluster Service, so gateway traffic to web does not leave the cluster. See Gateway to Langfuse Web.
• Generates the two shared secrets into the <release>-app Secret: the service key that web and the gateway use to authenticate each other, and the private key web uses to sign the gateway's trace upload tokens.

kubectl create secret generic langfuse-gateway -n langfuse \
  --from-literal=ai-gateway-service-key="$(openssl rand -hex 32)" \
  --from-file=ai-gateway-jwt-private-key=<(openssl ecparam -name prime256v1 -genkey -noout)

langfuse:
  gateway:
    serviceKey:
      secretKeyRef:
        name: langfuse-gateway
        key: ai-gateway-service-key
    jwt:
      privateKey:
        secretKeyRef:
          name: langfuse-gateway
          key: ai-gateway-jwt-private-key

For the full list of values, see the chart README.

Public URL

Langfuse shows the gateway's public URL in the UI and in the setup snippets for coding agents. The chart sets it on web as LANGFUSE_AI_GATEWAY_PUBLIC_URL, in this order:

1. langfuse.gateway.publicUrl, if set.
2. Otherwise the first host in langfuse.gateway.ingress.hosts, over https:// when the gateway Ingress has TLS enabled or langfuse.nextauth.url starts with https://, and http:// otherwise.
3. Without either, web shows gateway.<host of NEXTAUTH_URL>.

Set langfuse.gateway.publicUrl explicitly when you expose the gateway in another way, such as your own Ingress, the Gateway API, or a LoadBalancer Service. The value must be an origin without a path, such as https://llm.example.com.

TLS

We strongly recommend serving the gateway over HTTPS:

• Every request carries a gateway API key, and request and response bodies contain full prompts and model outputs.
• Coding clients such as Claude Desktop expect an https:// gateway URL.

Like Langfuse Web, the gateway does not terminate TLS itself. Terminate TLS in front of it, at the Ingress controller or at a load balancer. The gateway's host needs its own DNS record and a certificate that covers it. The chart doesn't need cert-manager for the gateway: use it only if you already issue certificates with it, otherwise bring your own certificate or terminate TLS at a load balancer.

langfuse:
  gateway:
    ingress:
      enabled: true
      hosts:
        - gateway.langfuse.example.com
      tls:
        enabled: true
        secretName: langfuse-gateway-tls
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt

kubectl create secret tls langfuse-gateway-tls -n langfuse \
  --cert=gateway.crt --key=gateway.key

langfuse:
  gateway:
    ingress:
      enabled: true
      hosts:
        - gateway.langfuse.example.com
      tls:
        enabled: true
        secretName: langfuse-gateway-tls

langfuse:
  gateway:
    ingress:
      enabled: true
      className: alb
      hosts:
        - gateway.langfuse.example.com
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
        alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<region>:<account>:certificate/<id>
        alb.ingress.kubernetes.io/healthcheck-path: /ready
        alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=620

To verify, check that the gateway answers over HTTPS with a valid certificate:

curl -v https://gateway.langfuse.example.com/health
# {"status":"ok"}

Gateway to Langfuse Web

The gateway calls Langfuse Web to resolve every request and to upload generations. Web's responses contain the provider API keys of your organization, so this connection must stay private or be encrypted.

By default, the chart points the gateway at web's in-cluster Service, http://<release>-web.<namespace>.svc.cluster.local:3000. This traffic stays inside your cluster network, avoids a round trip through the Ingress, and keeps request resolution fast. Because the URL is plain HTTP, the chart sets LANGFUSE_AI_GATEWAY_ALLOW_HTTP_WEB_URL=true on the gateway. Without that setting, the gateway only accepts https:// web URLs. For end-to-end encryption inside the cluster, use a service mesh with mutual TLS, as described in Encryption.

If the gateway cannot reach web's Service, for example because web runs in another cluster, set langfuse.gateway.webUrl to an https:// URL such as your NEXTAUTH_URL. Requests then pass through web's Ingress, so its body size limit must allow the gateway's requests. The chart's in-cluster default has no path, so if web runs under a custom base path, set langfuse.gateway.webUrl including the path, for example http://langfuse-web.langfuse.svc.cluster.local:3000/langfuse.

If web's certificate is signed by a private CA, mount the CA bundle with langfuse.gateway.pod.extraVolumes and langfuse.gateway.pod.extraVolumeMounts, and point SSL_CERT_FILE at it with langfuse.gateway.pod.additionalEnv.

Proxy timeouts and buffering

Model responses are streamed and can take minutes. The gateway ends a request after 600 seconds (LANGFUSE_AI_GATEWAY_MAX_EXECUTION_SECONDS) or after 120 seconds without progress (LANGFUSE_AI_GATEWAY_STALL_TIMEOUT_SECONDS). Everything between the client and the gateway must allow this:

• Set proxy read and send timeouts, or the load balancer idle timeout, above 120 seconds. 620 covers the full 600 seconds.
• Turn off response buffering, so streamed tokens reach the client as they arrive.
• Allow request bodies of at least 16 MiB (LANGFUSE_AI_GATEWAY_MAX_REQUEST_BYTES).

The chart sets no proxy annotations by default, because they depend on your Ingress controller. See the examples in Deploy with Helm for ingress-nginx and in TLS for the AWS Load Balancer Controller.

Operations

Scaling

The gateway is stateless, so scale it horizontally by adding pods. On Langfuse Cloud, we run several gateway instances per region and scale them on CPU utilization.

• Run at least two replicas in production, so that restarting a pod or a node doesn't interrupt all traffic. The chart keeps all but one pod running during node maintenance.
• Enable the HorizontalPodAutoscaler with langfuse.gateway.hpa.enabled: true and scale on CPU. The chart defaults (langfuse.gateway.hpa.minReplicas: 1, langfuse.gateway.hpa.maxReplicas: 2) are a starting point; raise them for production traffic.
• Size memory with the memory budget. Each pod reserves memory for every running request from LANGFUSE_AI_GATEWAY_MEMORY_BUDGET_BYTES (1 GiB by default), and returns 503 right away when the budget is used up. Set the pod's memory limit to at least the budget plus 512 MiB. To serve more concurrent or larger requests per pod, raise the budget and the limit together, or add pods.
• Remove replicas slowly when scaling down, so long-running streaming requests aren't cut off. By default, Kubernetes waits 5 minutes of lower load before it removes replicas; don't shorten that. Requests still running on a stopping pod get up to 5 minutes to finish, see Graceful shutdown.
• Scale Langfuse too. Every request makes web resolve the gateway key (cached for 60 seconds), and every generation is ingested like other traces. Plan web and worker capacity for the added trace volume, see Scaling.

To watch the gateway, set OTEL_EXPORTER_OTLP_ENDPOINT on it. It then exports operational traces and metrics, including gateway.memory.reserved.peak and gateway.admission.rejected, to your OpenTelemetry collector.

Health checks

The gateway serves two unauthenticated endpoints on port 8080, which the chart uses as Kubernetes probes:

/ready does not call web or the providers, so it stays 200 when they are unreachable. Those failures show up as errors on individual requests and in the gateway logs. Use /ready for load balancer health checks.

Graceful shutdown

Deployments and scaling down stop gateway pods while clients may still be streaming. The chart configures the pod so that running streams can finish:

1. langfuse.gateway.preStopSleepSeconds (10 s): the pod keeps serving while Kubernetes removes it from the Service and Ingress, so no new requests arrive.
2. langfuse.gateway.connectionDrainTimeoutSeconds (300 s): after SIGTERM, the gateway rejects new requests, reports 503 on /ready, and lets requests that are still running finish.
3. langfuse.gateway.telemetryShutdownTimeoutSeconds (30 s): after that, the gateway uploads the remaining generations to Langfuse.
4. langfuse.gateway.terminationGracePeriodSeconds (350 s): Kubernetes kills the pod after this time. It must be greater than the sum of the three steps above; the chart rejects values that are not.

Rotating keys

Web accepts a previous service key and a previous JWT public key next to the current ones. Rotate in three deployments so that every web pod accepts both keys while pods restart. Set the _PREVIOUS variables on web with langfuse.web.pod.additionalEnv, and pin the current keys with secretKeyRef as shown in Deploy with Helm.

Service key, shared by web and the gateway:

1. On web, set the new key as LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS and redeploy web. Web now accepts both keys.
2. Switch langfuse.gateway.serviceKey to the new key, which changes it on web and the gateway, and in the same deployment set LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS to the old key.
3. Remove LANGFUSE_AI_GATEWAY_SERVICE_KEY_PREVIOUS and redeploy web.

JWT signing key, used only by web:

1. On web, set the new public key as LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_PUBLIC_KEY and redeploy web. Derive a public key with openssl ec -in private-key.pem -pubout.
2. Switch langfuse.gateway.jwt.privateKey to the new private key, and in the same deployment set LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_PUBLIC_KEY to the old public key.
3. Wait at least 15 minutes, the lifetime of upload tokens signed with the old key. Then remove the previous public key and redeploy web.

Key IDs are derived from the public keys. Set LANGFUSE_AI_GATEWAY_JWT_PREVIOUS_KEY_ID only if you set LANGFUSE_AI_GATEWAY_JWT_KEY_ID explicitly.

Configuration reference

The chart sets these variables for you. You only need them when you deploy the gateway without the chart or override the chart's values.

Gateway container

Set additional gateway variables with langfuse.gateway.pod.additionalEnv. The global langfuse.additionalEnv does not apply to the gateway.

Web container

Troubleshooting

When you report a problem, include the langfuse-request-id response header of the failing request. For problems with the gateway in your organization's setup, see the AI Gateway troubleshooting.

If you experience any issues when self-hosting Langfuse, please:

1. Check out Troubleshooting & FAQ page.
2. Use Ask AI to get instant answers to your questions.
3. Ask the maintainers on GitHub Discussions.
4. Create a bug report or feature request on GitHub.