AI Gateway

AI is being used in many different areas of a company, with engineers using coding agents, often dozens of AI apps running in production, internally or externally. Companies need to keep visibility and control over rising AI usage, answering questions like:

• Who in our organization uses which models?
• What do we use it for?
• What does it cost?
• Is it secure?
• How can we optimize it?

The Langfuse AI Gateway is a central chokepoint to govern all AI traffic, easy to host, adopt, and enforce. It can log all requests natively in Langfuse to analyze and make decisions based on AI usage.

If you have feedback, questions, or feature requests, please share them on GitHub.

What the Langfuse AI Gateway can do

Most gateways tell you what AI costs. Because every call through the Langfuse AI Gateway becomes a full Langfuse trace, it's more powerful than that.

• See your spend. Cost and usage for every call land in one AI spend dashboard, by model and provider.
• Know who spends it. Every key belongs to a person or service and can carry a team or cost center, so AI spend breaks down the way your company is organized.
• Understand what it's spent on. Each call can be logged as a complete trace. You can see what an application or a team actually did with a model, and use the same search, evaluations, and annotation you already use in Langfuse. Or simply ask Halo: "What does this team use Claude Code for?"
• Control model usage. Approve models centrally, so expensive or unvetted ones don't creep in unnoticed.
• Incrementally adopt deeper tracing. Enforce logging every model call by mandating the gateway, and allow teams to incrementally add more detail (for example, tool call executions of complex agents) to AI traces in Langfuse via OpenTelemetry or one of our 50+ integrations. OpenTelemetry distributed tracing merges these traces within Langfuse.
• Use Langfuse as memory for agents. All data in Langfuse is available via MCP and the CLI, so you can give agents access to search all AI requests they made in the past, which is the foundation for self-improvement loops.
• Export metrics and raw logs to a data lake. This enables any custom downstream process in finance and data teams.

Roadmap

This is only the beginning. We have a strong roadmap for the AI Gateway to help govern AI usage:

• Cost: hard spend caps
• Reliability: fallbacks between providers
• Safety: guardrails, PII redaction
• Access: user provisioning (SCIM)
• More providers: Bedrock, Vertex, Google AI Studio, and custom providers that support the gateway's API formats
• Model routing: automatically pick the best model for the task

If you are interested in being a design partner on one of these, reach out to us.

Getting Started

Depending on your role, you're likely looking to:

• set up the gateway for your organization (admin)
• start using your organization's gateway to send your LLM calls (user)

For more information on how the gateway, keys, and model routing fit together, read Core Concepts.

---

---