Langfuse launches AI Gateway Learn more
IntegrationsSarvam AI

Observability for Sarvam AI with Langfuse

This guide shows how to trace Sarvam AI models with Langfuse from two places: a TypeScript app built with the Vercel AI SDK, and a real-time voice agent built with LiveKit Agents.

What is Sarvam AI? Sarvam AI builds models for Indian languages: the sarvam-30b and sarvam-105b chat models, Saaras speech-to-text, Bulbul text-to-speech, and translation and transliteration APIs.

What is Langfuse? Langfuse is an open-source AI engineering platform. It helps teams trace LLM applications, debug issues, evaluate quality, and monitor cost and latency. Use Langfuse Cloud (hosted by Langfuse, free tier, no infrastructure to run) or self-host it.

This page shows two example integrations. Both use OpenTelemetry: the framework emits spans for each model call, and the Langfuse SDK exports them to your Langfuse project.

  • Vercel AI SDK: text generation, streaming, and tool calls with the sarvam-ai-sdk provider.
  • LiveKit Agents: voice agents that use Sarvam for speech-to-text, the LLM, and text-to-speech.

These are examples, not the only way to trace Sarvam. You can trace Sarvam models with any Langfuse integration for the framework you already use. For example, use the OpenAI SDK with Sarvam's OpenAI-compatible base URL https://api.sarvam.ai/v1, the Sarvam services in Pipecat, or a Sarvam model in LangChain.

Vercel AI SDK

The community-maintained sarvam-ai-sdk package adds Sarvam as a Vercel AI SDK provider. Langfuse captures every AI SDK call that uses it through the Langfuse Vercel AI SDK integration.

Install dependencies

Requires Node.js 22 or later and AI SDK 7.

npm install ai@7 sarvam-ai-sdk @langfuse/vercel-ai-sdk @langfuse/otel @langfuse/tracing @opentelemetry/sdk-trace-node zod

Each sarvam-ai-sdk minor version targets one AI SDK major version (0.4.x targets AI SDK 7). Keep ai on the major version your sarvam-ai-sdk release expects. See the version table.

Configure credentials

Get Langfuse API keys from your project settings in Langfuse Cloud or your self-hosted instance, and a Sarvam API key from the Sarvam dashboard.

.env
LANGFUSE_SECRET_KEY = "sk-lf-..."
LANGFUSE_PUBLIC_KEY = "pk-lf-..."
LANGFUSE_BASE_URL = "https://cloud.langfuse.com" # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

SARVAM_API_KEY = "..."

Register the Langfuse integration

Create an instrumentation.ts file that sets up the LangfuseSpanProcessor and registers the Langfuse integration with the AI SDK once at startup.

instrumentation.ts
import { registerTelemetry } from "ai";
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { LangfuseVercelAiSdkIntegration } from "@langfuse/vercel-ai-sdk";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

export const langfuseSpanProcessor = new LangfuseSpanProcessor();

const tracerProvider = new NodeTracerProvider({
  spanProcessors: [langfuseSpanProcessor],
});
tracerProvider.register();

registerTelemetry(new LangfuseVercelAiSdkIntegration());

Call a Sarvam model

Import instrumentation.ts before any other module, then use sarvam(...) as the model in any AI SDK function. propagateAttributes sets the trace name, user, and session on every span the call creates.

app.ts
import { langfuseSpanProcessor } from "./instrumentation";

import { generateText, stepCountIs, tool } from "ai";
import { propagateAttributes } from "@langfuse/tracing";
import { sarvam } from "sarvam-ai-sdk";
import { z } from "zod";

async function main() {
  const { text } = await propagateAttributes(
    {
      traceName: "sarvam-weather-assistant",
      userId: "user-123",
      sessionId: "session-456",
    },
    () =>
      generateText({
        model: sarvam("sarvam-105b"),
        prompt: "बेंगलुरु में आज मौसम कैसा है? हिंदी में जवाब दें।",
        tools: {
          getWeather: tool({
            description: "Get the current weather in a city",
            inputSchema: z.object({
              city: z.string().describe("The city to get the weather for"),
            }),
            execute: async ({ city }) => ({
              city,
              temperatureC: 27,
              condition: "partly cloudy",
            }),
          }),
        },
        stopWhen: stepCountIs(3),
        telemetry: { functionId: "sarvam-weather-assistant" },
      }),
  );

  console.log(text);
}

// Flush buffered spans before a short-lived script exits
main().finally(() => langfuseSpanProcessor.forceFlush());

Run it with --env-file so the keys in .env are loaded before instrumentation.ts runs:

npx tsx --env-file=.env app.ts

In a long-running server, the span processor exports spans in the background and you don't need forceFlush(). In serverless functions, such as a Next.js route handler on Vercel, call after(() => langfuseSpanProcessor.forceFlush()) from next/server so spans are sent before the function stops.

View the trace in Langfuse

Open your project in Langfuse. The trace has an agent observation for the generateText call, a sarvam-105b generation with input, output, and token usage for each model step, and a tool observation for getWeather.

Vercel AI SDK trace in Langfuse with two sarvam-105b generations and a getWeather tool call

Example trace in Langfuse

LiveKit Agents

The livekit-plugins-sarvam plugin provides Sarvam speech-to-text, LLM, and text-to-speech for LiveKit Agents. LiveKit Agents has built-in OpenTelemetry support, so you pass it a TracerProvider that the Langfuse SDK exports from. The LiveKit integration guide covers the same setup for other providers and for Node.js.

Install dependencies

Requires Python 3.10 or later. Connecting the agent to a room also requires a LiveKit server, either LiveKit Cloud or self-hosted.

pip install "livekit-agents[sarvam,silero]~=1.8" langfuse opentelemetry-sdk python-dotenv

Configure credentials

Add your Langfuse, Sarvam, and LiveKit credentials to a .env file next to your agent.

.env
LANGFUSE_SECRET_KEY = "sk-lf-..."
LANGFUSE_PUBLIC_KEY = "pk-lf-..."
LANGFUSE_BASE_URL = "https://cloud.langfuse.com" # 🇪🇺 EU region
# Other Langfuse data regions include 🇺🇸 US: https://us.cloud.langfuse.com, 🇯🇵 Japan: https://jp.cloud.langfuse.com and ⚕️ HIPAA: https://hipaa.cloud.langfuse.com

SARVAM_API_KEY = "..."

LIVEKIT_URL = "wss://<your-project>.livekit.cloud"
LIVEKIT_API_KEY = "..."
LIVEKIT_API_SECRET = "..."

Create a traced voice agent

setup_langfuse() creates a TracerProvider, passes it to LiveKit, and attaches the Langfuse exporter. should_export_span=lambda span: True exports LiveKit's speech-to-text, text-to-speech, and turn spans as well as the LLM calls. The room name becomes the Langfuse session ID, so all turns of one call are grouped together.

agent.py
from dotenv import load_dotenv
from langfuse import Langfuse
from opentelemetry.sdk.trace import TracerProvider

from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli, function_tool
from livekit.agents.telemetry import set_tracer_provider
from livekit.plugins import sarvam, silero

load_dotenv()


def setup_langfuse(ctx: JobContext) -> TracerProvider:
    tracer_provider = TracerProvider()
    # Group all turns of one call into a Langfuse session
    set_tracer_provider(tracer_provider, metadata={"langfuse.session.id": ctx.room.name})

    # Reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_BASE_URL
    Langfuse(
        tracer_provider=tracer_provider,
        # Export LiveKit's STT, TTS and turn spans, not only LLM spans
        should_export_span=lambda span: True,
    )
    return tracer_provider


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a friendly voice assistant. Reply in the language the user speaks, in one or two short sentences.",
        )

    @function_tool
    async def get_weather(self, city: str) -> dict:
        """Get the current weather in a city."""
        return {"city": city, "temperature_c": 27, "condition": "partly cloudy"}


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    tracer_provider = setup_langfuse(ctx)

    async def flush_traces():
        tracer_provider.force_flush()

    ctx.add_shutdown_callback(flush_traces)

    session = AgentSession(
        stt=sarvam.STT(language="unknown"),  # auto-detect the spoken language
        # Use the /v1 endpoint: the plugin's default (/v2) can repeat tool calls
        llm=sarvam.LLM(model="sarvam-105b", base_url="https://api.sarvam.ai/v1"),
        tts=sarvam.TTS(target_language_code="hi-IN", model="bulbul:v3"),
        vad=silero.VAD.load(),
    )
    await session.start(agent=Assistant(), room=ctx.room)


if __name__ == "__main__":
    cli.run_app(server)

Run the agent

Download the voice activity detection model files once, then start the agent and talk to it in your terminal:

python agent.py download-files
python agent.py console

To connect it to a LiveKit room instead, run python agent.py dev and join the room from the Agents Playground.

View the trace in Langfuse

Open your project in Langfuse. Each conversation turn is an agent_turn observation with spans for speech-to-text and text-to-speech, sarvam-105b generations with token usage, and a tool observation for get_weather. Open Sessions to see all traces of one call together.

LiveKit agent trace in Langfuse with sarvam-105b LLM requests and a get_weather tool call

Example trace in Langfuse

Track cost

Langfuse records token usage for Sarvam models out of the box. To see cost, add a custom model definition for each Sarvam model you use, with Sarvam's per-token prices. Langfuse then calculates cost for every Sarvam generation, in both integrations.

Interoperability with the Python SDK

You can use this integration together with the Langfuse SDKs to add additional attributes to the observation.

The @observe() decorator provides a convenient way to automatically wrap your instrumented code and add additional attributes to the observation.

from langfuse import observe, propagate_attributes, get_client

langfuse = get_client()

@observe()
def my_llm_pipeline(input):
    # Add additional attributes (user_id, session_id, metadata, version, tags) to all spans created within this execution scope
    with propagate_attributes(
        user_id="user_123",
        session_id="session_abc",
        tags=["agent", "my-observation"],
        metadata={"email": "user@langfuse.com"},
        version="1.0.0"
    ):

        # YOUR APPLICATION CODE HERE
        result = call_llm(input)

        return result

# Run the function
my_llm_pipeline("Hi")

Learn more about using the Decorator in the Langfuse SDK instrumentation docs.

The Context Manager allows you to wrap your instrumented code using context managers (with with statements), which allows you to add additional attributes to the observation.

from langfuse import get_client, propagate_attributes

langfuse = get_client()

with langfuse.start_as_current_observation(
    as_type="span",
    name="my-observation",
    trace_context={"trace_id": "abcdef1234567890abcdef1234567890"},  # Must be 32 hex chars
) as observation:

    # Add additional attributes (user_id, session_id, metadata, version, tags)
    # to all observations created within this execution scope
    with propagate_attributes(
        user_id="user_123",
        session_id="session_abc",
        metadata={"experiment": "variant_a", "env": "prod"},
        version="1.0",
    ):
        # YOUR APPLICATION CODE HERE
        result = call_llm("some input")

# Flush events in short-lived applications
langfuse.flush()

Learn more about using the Context Manager in the Langfuse SDK instrumentation docs.

Troubleshooting

No observations appearing

First, enable debug mode in the Python SDK:

export LANGFUSE_DEBUG="True"

Then run your application and check the debug logs:

  • OTel observations appear in the logs: Your application is instrumented correctly but observations are not reaching Langfuse. To resolve this:
    1. Call langfuse.flush() at the end of your application to ensure all observations are exported.
    2. Verify that you are using the correct API keys and base URL.
  • No OTel spans in the logs: Your application is not instrumented correctly. Make sure the instrumentation runs before your application code.
Unwanted observations in Langfuse

The Langfuse SDK is based on OpenTelemetry. Other libraries in your application may emit OTel spans that are not relevant to you. These still count toward your billable units, so you should filter them out. See Unwanted spans in Langfuse for details.

Missing attributes

Some attributes may be stored in the metadata object of the observation rather than being mapped to the Langfuse data model. If a mapping or integration does not work as expected, please raise an issue on GitHub.

Next Steps

Once you have instrumented your code, you can manage, evaluate and debug your application:


Was this page helpful?

Last updated on