DocsSend your first request
DocsAI Gateway (beta)Get StartedSend your first request

Send your first request

Beta. The Langfuse AI Gateway is in public beta and can still change.

This page walks you through how to send your first request through the Gateway. If the gateway is not set up for your organization yet, ask an admin to follow Set up the gateway first.

Setup steps

Create a gateway API key

Go to Organization settings → AI Gateway → Gateway keys and select Create key.

Export the key and your region's gateway host in your shell. The setup for each client in the next steps uses these variables.

export LANGFUSE_GATEWAY_KEY="<your-gateway-key>"
# 🇪🇺 EU region
export LANGFUSE_GATEWAY_HOST="https://gateway.cloud.langfuse.com"
# 🇺🇸 US region: https://gateway.us.cloud.langfuse.com
# 🇯🇵 Japan region: https://gateway.jp.cloud.langfuse.com
# ⚕️ HIPAA region: https://gateway.hipaa.cloud.langfuse.com
# Self-hosted: the gateway URL of your deployment

You need the AI Gateway, Member, Admin, or Owner organization role. Viewers can't create gateway keys. See roles and permissions.

For CI or a shared service, ask an admin for a service account key. See API keys.

Pick a model

Copy the ID of a model you can call. Each ID has the form <credential>:<model>, for example openai-prod:<model>. See model IDs for which form to send.

Go to Organization settings → AI Gateway → Models to see the models you can call.

Gateway models page listing model IDs with a copy button, their provider, and supported routes

If the page says "No active providers", your organization hasn't added a provider credential yet. Ask your admin.

Call the gateway's models endpoints. A model list also confirms that your key works.

curl -s "$LANGFUSE_GATEWAY_HOST/openai/v1/models" \
  -H "Authorization: Bearer $LANGFUSE_GATEWAY_KEY" | head -c 300

curl -s "$LANGFUSE_GATEWAY_HOST/anthropic/v1/models?limit=1000" \
  -H "x-api-key: $LANGFUSE_GATEWAY_KEY" \
  -H "anthropic-version: 2023-06-01" | head -c 300

Connect your coding agent or app

Requires Claude Code 2.1 or later and an ANTHROPIC provider credential.

Log out of any Claude subscription, point Claude Code at the gateway's Anthropic route, and run a prompt:

claude /logout
export ANTHROPIC_BASE_URL="$LANGFUSE_GATEWAY_HOST/anthropic"   # no /v1: Claude Code appends /v1/messages
export ANTHROPIC_AUTH_TOKEN="$LANGFUSE_GATEWAY_KEY"            # sent as Authorization: Bearer
unset ANTHROPIC_API_KEY                                       # a different value here causes a 401
claude -p "Reply with: hello gateway"

Full Claude Code guide

Requires an ANTHROPIC provider credential.

  1. Open Developer → Configure Third-Party Inference. If the Developer menu is unavailable, open Help → Troubleshooting and enable Developer Mode.
  2. Under Connection, set Inference provider to Gateway and Gateway base URL to https://gateway.cloud.langfuse.com/anthropic, without /v1. Set Credential kind to Static API key, enter your gateway key under Gateway API key, and select Bearer as the Gateway auth scheme.
  3. Under Capabilities → General restrictions, add * to Allowed egress hosts. Otherwise, Claude can't reach services outside the gateway, such as GitHub.
  4. Select Apply Changes, then Save & Restart, and send a message in a new session.

This example uses the EU host. If your organization is in another region, use your region's gateway host.

Full Claude Desktop guide

Requires Codex CLI 0.157 or later and an OPENAI provider credential.

Export the gateway key:

 export LANGFUSE_GATEWAY_KEY="<your-gateway-key>"

Add this to ~/.codex/config.toml ($CODEX_HOME/config.toml if you set CODEX_HOME). Put model and model_provider next to your other top-level keys, before any [table] section, and the [model_providers.langfuse] table after them:

~/.codex/config.toml
model = "<model-id>"               # an OpenAI model ID
model_provider = "langfuse"

[model_providers.langfuse]
name = "Langfuse AI Gateway"
base_url = "https://gateway.cloud.langfuse.com/openai/v1"
env_key = "LANGFUSE_GATEWAY_KEY"   # sent as Authorization: Bearer
wire_api = "responses"

This example uses the EU host. If your organization is in another region, use your region's gateway host.

Then run a prompt:

codex exec "Reply with exactly: PONG"

Full Codex guide

Requires OpenCode 2.0 or later and an ANTHROPIC provider credential.

Point OpenCode's Anthropic provider at the gateway in opencode.json (project root or ~/.config/opencode/opencode.json). Remove any stored OpenCode account connection first, otherwise OpenCode keeps using it.

opencode.json
{
  "$schema": "https://opencode.ai/config.json",
  "model": "anthropic/<model-id>",
  "small_model": "anthropic/<small-model-id>",
  "providers": {
    "anthropic": {
      "settings": {
        "baseURL": "https://gateway.cloud.langfuse.com/anthropic/v1",
        "apiKey": "{env:LANGFUSE_GATEWAY_KEY}"
      }
    }
  }
}

This example uses the EU host. If your organization is in another region, use your region's gateway host.

Then run a prompt:

opencode run "Reply with exactly: pong"

Full OpenCode guide

Requires Pi 1.5.0 or later and an ANTHROPIC provider credential.

Install Pi:

npm install -g @earendil-works/pi-coding-agent

Add the gateway's Anthropic route as a provider in ~/.pi/agent/models.json. Set the model values to those of the model you picked:

~/.pi/agent/models.json
{
  "providers": {
    "langfuse-anthropic": {
      "baseUrl": "https://gateway.cloud.langfuse.com/anthropic",
      "api": "anthropic-messages",
      "apiKey": "$LANGFUSE_GATEWAY_KEY",
      "compat": { "sendSessionAffinityHeaders": true },
      "models": [
        {
          "id": "<anthropic-model-id>",
          "reasoning": true,
          "input": ["text", "image"],
          "contextWindow": 200000,
          "maxTokens": 64000
        }
      ]
    }
  }
}

This example uses the EU host. If your organization is in another region, use your region's gateway host.

Then run a prompt:

pi -p --provider langfuse-anthropic --model <anthropic-model-id> "Reply with exactly: pong"

Full Pi guide

Requires an ANTHROPIC provider credential.

Point Copilot CLI at the gateway's Anthropic route and start it:

export COPILOT_PROVIDER_BASE_URL="$LANGFUSE_GATEWAY_HOST/anthropic"
export COPILOT_PROVIDER_TYPE="anthropic"
export COPILOT_PROVIDER_API_KEY="$LANGFUSE_GATEWAY_KEY"
export COPILOT_MODEL="<model-id>"   # an Anthropic model ID
copilot

Send a prompt in copilot. If ~/.copilot/providers.json already lists providers or models, it takes precedence and these variables have no effect. For the OpenAI route and the GitHub Copilot app, see the full guide.

Full GitHub Copilot guide

Set the base URL to the gateway's OpenAI route and pass the gateway key as the API key.

Python:

pip install openai
import os

from openai import OpenAI

client = OpenAI(
    base_url=os.environ["LANGFUSE_GATEWAY_HOST"] + "/openai/v1",
    api_key=os.environ["LANGFUSE_GATEWAY_KEY"],
)

response = client.responses.create(model="<model-id>", input="Say hello")
print(response.output_text)

TypeScript:

npm install openai
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
  apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});

const response = await client.responses.create({
  model: "<model-id>",
  input: "Say hello",
});
console.log(response.output_text);

Full OpenAI SDK guide

Set the base URL to the gateway's Anthropic route, without /v1, and pass the gateway key as the API key.

Python:

pip install anthropic
import os

from anthropic import Anthropic

client = Anthropic(
    base_url=os.environ["LANGFUSE_GATEWAY_HOST"] + "/anthropic",
    api_key=os.environ["LANGFUSE_GATEWAY_KEY"],
)

message = client.messages.create(
    model="<model-id>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Say hello"}],
)
print(message.content[0].text)

TypeScript:

npm install @anthropic-ai/sdk
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/anthropic`,
  apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});

const message = await client.messages.create({
  model: "<model-id>",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Say hello" }],
});
console.log(message.content);

Full Anthropic SDK guide

Point the AI SDK's OpenAI provider at the gateway's OpenAI route:

npm install ai @ai-sdk/openai
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const openai = createOpenAI({
  baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
  apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});

const { text } = await generateText({
  model: openai("<model-id>"),
  prompt: "Say hello",
});
console.log(text);

For Anthropic models, see the full guide.

Full Vercel AI SDK guide

Call a model with the OpenAI Chat Completions API:

curl -si "$LANGFUSE_GATEWAY_HOST/openai/v1/chat/completions" \
  -H "Authorization: Bearer $LANGFUSE_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Say hello"}]}'

The response is OpenAI's own response. Its headers include langfuse-trace-id, the ID of the trace the gateway recorded.

REST API

Replace <model-id> with the model ID you picked in the previous step.

See your trace in Langfuse (optional)

Open the organization's ingestion project in Langfuse and go to Tracing to see your request. What's recorded depends on the organization's ingestion mode.

You need Viewer access or above to the ingestion project to see gateway traces. If you can't open it, ask an admin to add you.

Not seeing what you expected?

For error codes, see Troubleshooting. Quote the langfuse-request-id response header when you report a problem.

Was this page helpful?

Last updated on

PreviousSet up Gateway as an admin
NextConcepts

On this page

Send your first requestSetup stepsNot seeing what you expected?

Actions

Give us feedbackEdit this page on GitHub
GitHub