What are the known limitations of the AI Gateway beta?

• Input over 5 MiB is not recorded. When a request body exceeds 5 MiB, the generation has no input, and the metadata key langfuse.gateway.request.input_omitted says why (size_limit, with the limit in langfuse.gateway.request.input_limit_bytes). Output, usage, cost, and session are still recorded. Very long Codex and Claude Code sessions can hit this.
• Output over 1 MiB is recorded partially. The response still reaches the client in full, but the recorded output is cut and marked incomplete.
• Cost bugs, fixes in progress:
  • Failed and cancelled calls on the OpenAI route get estimated usage and cost. Long cancelled Codex calls showed about $6 each.
  • OpenAI cache_write_tokens are priced at $0.
  • Anthropic 1-hour cache writes are priced at the 5-minute rate, 37.5% too low.
• One trace per model call by default for Claude Code, OpenCode, and Pi. Only Codex groups a turn into one trace out of the box. Use the Sessions view, or set up turn grouping for Claude Code, OpenCode, or Pi.
• Generic names. Generations are named after the API format (anthropic.messages, openai.responses), and trace names default to the agent name.
• Duplicate metadata. attributes.user.id, attributes.session.id, and attributes.langfuse.trace.* appear in metadata next to the canonical fields.
• OpenCode: title calls can't be told apart from agent steps, and subagents get a separate session.
• Limits: 16 MiB request body, 600 seconds per call, and 120 seconds without progress. On Langfuse Cloud, requests are also rate limited per organization. Self-hosted deployments can change the request limits and have no rate limits, see Self-hosting the AI Gateway.
• No retries, fallbacks, or caching.
• Not served: websockets and realtime APIs.
• Decision models aren't supported. The gateway doesn't serve the OpenAI Decisions API (/v1/decisions) or TypeSafe's System One API, and it has no TypeSafe provider credential. Requests to these endpoints return 404. Decision model evaluators keep using their own LLM connection. Structured outputs (response_format, JSON schemas) on Chat Completions and Responses are passed through as usual.
• Untested: Anthropic server tools (web search and fetch) and interactive Codex /compact.

Rate limits on Langfuse Cloud

During the beta, each organization has two per-minute limits on Langfuse Cloud: the number of requests, and the total size of request bodies. They apply across all keys and projects of the organization.

Over a limit, the gateway returns 429 with a Retry-After header, in the provider's usual error format, so SDKs and coding agents back off as they do for provider rate limits. The error message says the limit comes from Langfuse.

These limits are subject to change during the beta. If you need higher limits, contact us.

---

---