--- title: "Evaluating and Monitoring Voice AI Agents" date: 2025/01/22 description: "Learn how to use Langfuse in your Voice AI development workflow for comprehensive voice agent testing and optimization." ogImage: /images/blog/2025-01-22-evaluating-voice-ai-agents/evaluating-voice-ai-agents.png tag: guide, evaluation author: Marc --- As Voice AI applications continue to advance, developers are met with complex challenges in testing, evaluating, and monitoring their voice agents. In this blog post, we'll explore how you can create more robust and reliable voice applications. We'll draw insights from setups we have seen from our users at [Langfuse](/) and the recent discussion ([full video](https://www.youtube.com/watch?v=hPrPqry1yQQ)) between me and Brooke (Co-Founder of [Coval](https://www.coval.dev/)) to provide a comprehensive guide on Voice AI evaluation. ## The Evolution of Voice AI Testing Voice AI applications present complexities that extend beyond traditional LLM implementations. In addition to [challenges](/faq/all/llm-observability) caused by the non-deterministic nature of language models, developers must also handle: - Audio Quality and Metrics - User Interruptions - Speech-to-Text (STT) Accuracy - Text-to-Speech (TTS) Output Quality - Real-Time Streaming Interactions As voice applications mature, the need for both high-level integration testing and detailed component evaluation becomes critical. ## Understanding the Voice AI Testing Pyramid Developing effective voice applications requires a dual approach to evaluation strategies: - **Online Evaluation:** Focuses on real-time production monitoring, performance tracking, and analyzing user interactions. - **Offline Evaluation:** Involves development testing, ranging from end-to-end agent testing to granular unit tests and validating conversation flows. Using this testing pyramid is essential for effective Voice AI testing, ensuring your voice agents perform optimally in live environments. ## Evaluations of Single Messages vs. Conversation Level The second duality we see in voice agent evaluation is between observing and evaluating single messages and evals on the whole conversation. **Single turn evaluations:** - [Trace](https://langfuse.com/docs/tracing) the step-by-step execution of a single message - Monitor the tool calls and other application logic used by the voice agent - Stream-based interaction analysis **Multi turn evaluations:** - Performing end-to-end simulation testing on the whole conversation - Testing for regressions caused by different prompt versions or model changes - Classifying and detecting anomalies in the conversation flow ## Integration Best Practices and Development Workflow Usually, there are two phases in the voice agent development workflow: **Early development stages:** - Quick integration tests and online evaluations - Trace and debug individual components of the conversation **Application running in production:** - Implement specific unit tests for cases spotted in development - Detailed performance monitoring and conversation level evaluations - Ongoing regression testing The type of evaluation also depends on the type of the voice application. Some applications might require a closer monitoring of model costs whereas other might focus on the conversation flow and the accuracy of tool calls: **Transactional Voice Applications (e.g., Appointment Scheduling):** - Trace individual function calls and apply evaluations to single messages. - Perform end-to-end testing of complete user journeys. **Complex Applications (e.g., Virtual Assistants):** - Focus on conversation-level testing and monitor conversation arcs. - Monitor tool calls and application logic. We are excited that [Coval](https://www.coval.dev/) will natively integrate with Langfuse. With this integration, Langfuse users can use Coval to perform end-to-end simulation testing on the whole conversation of their voice agents. Reach out if you are interested to try it. ## Resources - Watch the full discussion with Brooke Hopkins and Marc Klingen [here](https://www.youtube.com/watch?v=hPrPqry1yQQ). - Learn more about Langfuse: - [Tracing](https://langfuse.com/docs/tracing) LLM applications - [LLM Observability Challenges](/faq/all/llm-observability) - Tracking [model Usage and Cost](https://langfuse.com/docs/model-usage-and-cost) - Performing [LLM-as-a-Judge Evaluations](https://langfuse.com/docs/scores/model-based-evals) - Check out the [Coval docs](https://docs.coval.dev/getting_started/welcome). --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/blog/2025-01-22-evaluating-voice-ai-agents.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .