---
title: "Catching conversation signals in Langfuse"
date: 2026/09/23
description: "Score every chat agent conversation for frustration, corrections, follow-ups, and resolution — in one Jev pass, at a fraction of LLM-as-a-judge cost."
tag: engineering
author: Annabell
highlight: true
---
Chat agents can handle millions of user interactions every day. A volume, impossible for a human to review manually. Jev-like models now open an opportunity to review production traces for signals of failure at scale. Something that can get costly with LLMs and is hard to solve with [deterministic code-based evals](/docs/evaluation/evaluation-methods/code-evaluators).
## About user signals [#about-user-signals]
There are different types of signals in chat applications that can be used to monitor and improve AI applications.
- You can ask for [**explicit user feedback, like thumbs up or down**](/docs/observability/features/user-feedback)
- You can **track behavior**, like dismissing a chat
- You can **track factual outcomes of a chat**, like a completed booking
- You can **track conversation signals**, like a user being upset or asking follow ups.
Especially **conversation signals** help you steer your attention to the relevant traces from production as soon as manual review is not feasible anymore. They tell you something is going wrong. See the table below for a typical list of conversation signals.
Established approaches to catch these signals were especially centered around [LLM-as-a-judge](/docs/evaluation/evaluation-methods/llm-as-a-judge). With scale, the caused costs were primarily contained through [sampling](/docs/observability/features/sampling). Jev-like models now allow a full pass on all production traces.
For more on the signal taxonomy (explicit, behavioral, conversation, and
outcome), see [Capturing signals](/academy/monitoring/capturing-signals) in
the Langfuse Academy. To set up Jev evaluators in Langfuse, see [Jev as a
judge](/docs/evaluation/evaluation-methods/jev-as-a-judge).
## Catching conversation signals with Jev [#catching-conversation-signals-with-jev]
Jev-like models offer the advantage that the same state or context can be reused to answer multiple questions in parallel.
This means the same conversation can concurrently be evaluated for follow-up questions, user frustration, a user disagreeing with the agent, and if the user seems to be happy with the outcome.
Let us look at an example conversation in a support resolution. The user wants to rebook a flight. Over the course of the interaction she has to first correct the AI, then ask a follow up question, but then finally gets the request resolved. Multiple things are true here. The conversation outcome was successful. Nevertheless, the AI made a mistake first, and was apparently not effective enough in their answers to not cause a follow-up by a user.
With Jev, these signals can be caught concurrently in one pass, at the fraction of the cost of using frontier LLM models. All dimensions can be modelled as individual [noul questions](https://docs.typesafe.ai/primitives/noul), to get separate verdicts on the different conversation signals.
## Using Jev in Langfuse [#using-jev-in-langfuse]
While evals should be custom to your application, catching conversation signals is universal. The Langfuse template gallery now offers a full Jev based batch evaluator template to catch production signals.
### Go to the Evaluator tab
Go to the [Evaluator tab](https://cloud.langfuse.com/project/~/evals) and click on New evaluator.
### Choose the User Conversation Signal template
Choose the ‘User Conversation Signal’ template.
### Add your TypeSafe connection
Add your [Jev connection](/docs/evaluation/evaluation-methods/jev-as-a-judge#add-a-typesafe-connection) if you have not already.
### Choose the observations to run on and test
Choose the observations to run on and test.
### Save and set rule for evaluators to run on
Save and set rule for evaluators to run on.
The scores will then appear on incoming traces. You can also opt to do a pass on historic traces. To visualize the scores you can create [dashboards](/docs/metrics/features/custom-dashboards), and create [saved views](/changelog/2025-05-20-save-table-views) in your tracing table.
## Ready to get started with Langfuse?
Join thousands of teams building better LLM applications with Langfuse's open-source observability platform.
- [Start free](/cloud)
- [Documentation](/docs)
- [Talk to an expert](/talk-to-us)
---
## Agent Instructions
This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/blog/2026-09-23-catching-conversation-signals-in-langfuse.md`.
### Querying these docs
If the answer is not on this page, query the documentation instead of guessing:
- **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question:
```bash
curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?"
```
- **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt).
### Before writing Langfuse code
- **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results.
- **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application.
- **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples.
- **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx @langfuse/cli api `) to read or write traces, prompts, datasets, and scores from the terminal.
Found an error in these docs? Please open an issue at .