OpenAI SDK with the Langfuse AI Gateway
Beta. The Langfuse AI Gateway is in public beta and can still change.
Use the official OpenAI SDKs for Python and TypeScript with the gateway by changing the base URL and the API key. Requests need an OPENAI provider credential in your organization.
For tracing the OpenAI SDK without the gateway, see the OpenAI (Python) and OpenAI (JS/TS) integrations.
Before you start, create a gateway key and export LANGFUSE_GATEWAY_HOST and LANGFUSE_GATEWAY_KEY as in Send your first request. Replace <model-id> with an ID from the models endpoint.
Configure the client
Set the base URL to the gateway's OpenAI route, /openai/v1, and pass the gateway key as the API key. The SDK sends it as Authorization: Bearer.
pip install openaiimport os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["LANGFUSE_GATEWAY_HOST"] + "/openai/v1",
api_key=os.environ["LANGFUSE_GATEWAY_KEY"],
)npm install openaiimport OpenAI from "openai";
const client = new OpenAI({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});Responses API
response = client.responses.create(
model="<model-id>",
input="Say hello",
)
print(response.output_text)const response = await client.responses.create({
model: "<model-id>",
input: "Say hello",
});
console.log(response.output_text);Chat Completions
completion = client.chat.completions.create(
model="<model-id>",
messages=[{"role": "user", "content": "Say hello"}],
)
print(completion.choices[0].message.content)const completion = await client.chat.completions.create({
model: "<model-id>",
messages: [{ role: "user", content: "Say hello" }],
});
console.log(completion.choices[0].message.content);Streamed Chat Completions end with a usage chunk whose choices array is
empty, even if you didn't ask for usage. Skip chunks without choices. See
streaming usage.
Add request headers
Set request headers on the client for every call, or per call for values such as a session ID.
client = OpenAI(
base_url=os.environ["LANGFUSE_GATEWAY_HOST"] + "/openai/v1",
api_key=os.environ["LANGFUSE_GATEWAY_KEY"],
default_headers={"langfuse-tags": "checkout,production"},
)
response = client.responses.create(
model="<model-id>",
input="Say hello",
extra_headers={"langfuse-session-id": "<session-id>"},
)
completion = client.chat.completions.create(
model="<model-id>",
messages=[{"role": "user", "content": "Say hello"}],
extra_headers={"langfuse-session-id": "<session-id>"},
)const client = new OpenAI({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
defaultHeaders: { "langfuse-tags": "checkout,production" },
});
const response = await client.responses.create(
{ model: "<model-id>", input: "Say hello" },
{ headers: { "langfuse-session-id": "<session-id>" } },
);
const completion = await client.chat.completions.create(
{
model: "<model-id>",
messages: [{ role: "user", content: "Say hello" }],
},
{ headers: { "langfuse-session-id": "<session-id>" } },
);The gateway already records every call. Don't also trace the same client with the Langfuse OpenAI integration, or each call is recorded twice.
Last updated on
