Vercel AI SDK with the Langfuse AI Gateway
Beta. The Langfuse AI Gateway is in public beta and can still change.
Use the AI SDK by Vercel with the gateway by pointing its OpenAI or Anthropic provider at the gateway. Each provider needs the matching provider credential in your organization: OPENAI for OpenAI models, ANTHROPIC for Anthropic models.
For tracing the Vercel AI SDK without the gateway, see the Vercel AI SDK integration.
Before you start, create a gateway key and export LANGFUSE_GATEWAY_HOST and LANGFUSE_GATEWAY_KEY as in Send your first request. Replace <model-id> with an ID from the models endpoint.
Configure the provider
Create the provider with the gateway's route as base URL and the gateway key as API key.
Use the gateway's OpenAI route, /openai/v1, with the OpenAI provider.
npm install ai @ai-sdk/openaiimport { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";
const openai = createOpenAI({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});
const { text } = await generateText({
model: openai("<model-id>"),
prompt: "Say hello",
});
console.log(text);openai("<model-id>") uses the Responses API. To use Chat Completions instead, use openai.chat("<model-id>").
Use the gateway's Anthropic route with the Anthropic provider. Unlike the Anthropic SDK, the AI SDK provider expects /v1 in the base URL, so use /anthropic/v1.
npm install ai @ai-sdk/anthropicimport { createAnthropic } from "@ai-sdk/anthropic";
import { generateText } from "ai";
const anthropic = createAnthropic({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/anthropic/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
});
const { text } = await generateText({
model: anthropic("<model-id>"),
prompt: "Say hello",
});
console.log(text);Add request headers
Set request headers on the provider for every call, or per call for values such as a session ID.
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";
const openai = createOpenAI({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/openai/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
headers: { "langfuse-tags": "checkout,production" },
});
const { text } = await generateText({
model: openai("<model-id>"),
prompt: "Say hello",
headers: { "langfuse-session-id": "<session-id>" },
});import { createAnthropic } from "@ai-sdk/anthropic";
import { generateText } from "ai";
const anthropic = createAnthropic({
baseURL: `${process.env.LANGFUSE_GATEWAY_HOST}/anthropic/v1`,
apiKey: process.env.LANGFUSE_GATEWAY_KEY,
headers: { "langfuse-tags": "checkout,production" },
});
const { text } = await generateText({
model: anthropic("<model-id>"),
prompt: "Say hello",
headers: { "langfuse-session-id": "<session-id>" },
});The gateway already records every call. Don't also trace the same calls with the Langfuse Vercel AI SDK integration, or each model call is recorded twice.
Last updated on
