--- title: Customer support chatbot sidebarTitle: Support chatbot description: How a customer-facing support chatbot is rolled out in phases, from offline testing on historical tickets to internal drafts to answering customers directly. --- # Customer support chatbot This is an example to illustrate the concepts of the [Langfuse Academy](/academy). ## Context A SaaS company wants to automate its customer support chat. Today a support team answers every conversation, and years of resolved tickets sit in the ticketing system. The bot would speak to customers in the company's name, so a bad answer does damage beyond one conversation: customers lose trust in support as a whole. That risk rules out shipping straight to customers. Instead, the rollout happens in three phases, each one earning the confidence for the next: | Phase | What's live | Gate to the next phase | | ------------------ | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | 1. Offline | Nothing, the agent is in development | Drafts good enough that the support team wants to use them | | 2. Internal | The agent drafts replies, the support team edits and sends | The difference between drafted and sent replies stays small across ticket categories, and the qualitative feedback from the support team is good | | 3. Customer-facing | The bot answers customers directly | Nothing, this is the end goal: the agent keeps being improved continuously in this phase | By employing this staged approach, we mitigate a lot of risk while getting production signals early. ## Phase 1: creating the initial agent version In order for the support team to actually use the internal agent, we need to make sure the initial version is already useful for them. To get there, we can lean on a lot of historical data from resolved tickets that we already have, no synthetic data generation needed. **The AI engineering loop** - **Trace** — not live yet - **Monitor** — not live yet - **Build datasets** — from resolved tickets - **Experiment** — prompt and retrieval variants - **Evaluate** — compare against the human reply We can use AI to understand which patterns exist in the historical data from the support ticketing system, and use that to group into a few datasets by recurring behavior: **Dataset: howto-questions** Questions about how to use a product feature. The reply explains the steps and links the relevant help article. | Input | Expected output | | --- | --- | | How do I set up recurring tasks? | Refers to the 'Recurring tasks' help article. Recommends setting a repeat schedule in the task menu. | | Can I export my data to CSV? | Refers to the 'Exporting data' article. Confirms CSV export is available on the Pro plan. | **Dataset: account-changes** Requests that touch the customer's plan, seats, or roles. The reply points at the right settings and who can change them. | Input | Expected output | | --- | --- | | I can't add my colleague to our workspace, the invite says 'seat limit reached'. | Refers to the seat management docs. Recommends deactivating a former member or adding seats under Billing. | | How do I transfer ownership of a workspace? | Refers to the 'Workspace roles' docs. Recommends transferring ownership under Settings → Members, which requires an admin. | **Dataset: troubleshooting** Something does not work the way the customer expects. The reply diagnoses the issue or asks for what's needed to diagnose it. | Input | Expected output | | --- | --- | | The CSV export is missing the custom fields we added. | Refers to the export settings docs. Recommends selecting 'Export with custom fields' in the export dialog. | | My calendar integration stopped syncing this morning. | Asks for the integration name and the last successful sync time, and links the status page. | Splitting into multiple datasets vs keeping it in one is a [trade-off](/academy/datasets#what-makes-a-good-dataset). Grouping allows you to measure performance per use case and lets you run a subset when needed. One big dataset can make sense too, especially early on when the optimal split isn't clear yet. We now iterate on the prompt, [running each version against these datasets](/academy/experiments#how-experiments-are-used), until the drafts reach the bar where the support team would actually find them useful. Each run is graded with: **Evaluators** - `resolution_match` — LLM-as-a-judge, binary, per dataset item - Does the draft refer to the same sources and recommend the same thing as the human reply did? - A [reference-based](/academy/evaluate#reference-based-vs-reference-free) evaluator: the expected output gives it the sources and recommendation from the human reply to compare against. It has to be an LLM-as-a-judge rather than a code check because two replies can recommend the same thing in completely different words. It needs the expected output to run, so it can grade experiment runs on these datasets but never live traffic. - `links_valid` — Code, binary, per dataset item - Every help article linked in the draft exists in the help center - Added after manual review turned up drafts linking help articles that don't exist. Whether an article exists is a lookup in the help-article index, so a [code evaluator settles it exactly](/academy/evaluate/writing-evaluators#make-it-verifiable), in milliseconds and for free. It also needs no expected output, so the same check can keep running on live replies in the later phases. - `manual review` — Human - Vibe check on a sample of drafts Notice that we are not judging the tone of voice automatically here. This is on purpose: a few [manual checks](/academy/evaluate#how-evaluation-typically-evolves) are enough to make sure the drafts sound acceptable, and we will use phase 2 to get the tone of voice perfectly aligned. When the evaluator scores are good enough and the manual review looks good, the agent moves into the support ticketing system. Before moving on to the next section, this is what a trace could look like: **The observations inside a draft-support-reply trace** - [Trace] `draft-support-reply` (dataset item: account-changes, 3.1s) - Input: "I can't add my colleague to our workspace, the invite says 'seat limit reached'." - Output: "Hi Sam, all 10 seats on your plan are in use. You can free one up by deactivating a former member under Settings → Members, or add seats under Billing." - [Retriever] `find-similar-tickets` (0.5s) - Input: the ticket text - Output: 3 resolved tickets: T-3107, T-2954, T-2381 (seat limit, adding members) - [Retriever] `search-help-articles` (0.4s) - Input: "seat limit, add seats" - Output: 'Managing seats' · 'Billing settings' - [Tool] `fetch-account-details` (0.3s) - Input: workspace W-2209 - Output: plan: Team · seats: 10 of 10 in use - [Gen] `draft-reply` (gpt-4.1, 1.4k tok, $0.01, 1.9s) - Input: the ticket text · account details · 2 help articles · 3 resolved tickets - Output: "Hi Sam, all 10 seats on your plan are in use. You can free one up by deactivating a former member under Settings → Members, or add seats under Billing." ## Phase 2: internal rollout In this phase, the agent drafts a reply for every incoming ticket, and a support team member can use it and edit it before sending. We can use the difference between the agent's draft and what the team member actually sends as a very reliable [user signal](/academy/monitoring#user-feedback). Another, more explicit user signal is a thumbs up or thumbs down button on the draft, where the team can also leave written feedback. **The AI engineering loop** - **Trace** — every draft is traced - **Monitor** — edit_type on every trace - **Build datasets** — heavily edited drafts become new items - **Experiment** — fixes for recurring edit patterns - **Evaluate** — rerun the datasets before shipping Both signals are captured as [scores](/docs/evaluation/scores/overview) on every draft [trace](/academy/tracing#anatomy-of-a-trace): **Evaluators** - `edit_type` — LLM-as-a-judge, categorical, on the trace - Classifies the human edit as `none`, `tone`, `corrected`, or `added` - A judge comparing the draft to the sent reply. `tone` means the team member reworded it but kept the content, `corrected` that something wrong was removed or fixed, `added` that the draft was kept but missing information was added. It has to be a judge because telling a rewording apart from a content change requires reading both texts. Like any judge, it starts out untested: hand-label a sample of drafts and their sent replies, and [check that the judge classifies them the way you would](/academy/evaluate/writing-evaluators#validate-the-judge). - `human_feedback` — Human, binary plus comment, on the trace - Thumbs up or down on the draft, with optional written feedback - Captured directly from the button in the ticketing system, no evaluator needed: the click writes the score. The written comments give context the diff alone can't. Traces with a bad score can then be looked at, improved, and added to the datasets if they weren't covered before. One way to do this is using the [error analysis](/academy/monitoring/error-analysis) process. One iteration through the loop then looks like this: [monitoring](/academy/monitoring#metrics-and-signals) shows a spike of `corrected` drafts in the `troubleshooting` category. The traces reveal the agent keeps citing an outdated export dialog, because the help-article index is stale. The index gets refreshed, the `troubleshooting` dataset is rerun to confirm the fix, and the share of `corrected` drafts drops. No new evaluator comes out of this: a stale index is a [one-time fix](/academy/evaluate/choosing-what-to-evaluate#fix-first), not a failure mode to keep tracking. The share of edited drafts should go down over time. Once drafts go out mostly unchanged across all ticket categories, the agent is ready to face customers. ## Phase 3: customer-facing rollout The bot now answers customers directly, and the signal that drove phase 2 disappears: nobody edits the reply before the customer sees it. Instead, we introduce a couple of new implicit user signals that will help us learn and improve over time. Instead of going full-auto immediately, you could also automate only the request categories that cleared the phase 2 gate, while the rest keeps going through the support team for a little longer. **The AI engineering loop** - **Trace** — every customer conversation - **Monitor** — user signals and risk monitoring - **Build datasets** — bad conversations become new items - **Experiment** — fixes for what monitoring surfaces - **Evaluate** — rerun the datasets before shipping Customers rarely rate their support chat, so the main focus is on gathering implicit user signals: **Evaluators** - `handoff_requested` — LLM-as-a-judge, binary, on the trace - Detects the customer asking for a human - Customers ask for a human in free text, so there is nothing for a code evaluator to match on. Scoring the trace puts the flag on the exact turn where the customer gave up on the bot. - `question_repeated` — LLM-as-a-judge, binary, on the trace - Detects the customer rephrasing the same question - A form of [implicit feedback](/academy/monitoring#user-feedback): a customer repeating themself means the previous answer didn't land. Only a judge reading the conversation can tell a rephrasing apart from a follow-up question. - `session_outcome` — LLM-as-a-judge, categorical, on the session - Classifies how the conversation ended - Runs when the conversation closes and classifies it as `resolved`, `abandoned`, or `handed_off`. The three endings each mean something different for the bot's performance and are mutually exclusive, so this is [one categorical evaluator rather than several overlapping binary ones](/academy/evaluate/writing-evaluators#binary-verdicts). Scored on the session rather than the trace, since the outcome belongs to the conversation as a whole. With no human in the loop anymore, we monitor every reply for reputation-damaging behavior. These two evaluators come from [hard constraints](/academy/evaluate/choosing-what-to-evaluate#goals-and-hard-constraints) rather than observed failures: they exist from day one, even though neither has ever fired. They don't block anything: they flag sent replies so the team can follow up with the customer quickly. **Evaluators** - `data_leak` — LLM-as-a-judge, binary, on the trace - Does the reply reveal data about anyone other than this customer? - [Reference-free](/academy/evaluate#reference-based-vs-reference-free), so it can run on every production reply without an expected output. Recognizing personal data in any phrasing requires language understanding, hence an LLM-as-a-judge. - `out_of_scope_help` — LLM-as-a-judge, binary, on the trace - Does the reply promise a refund, discount, or commitment instead of handing off? - Also reference-free, running on every production reply. A judge rather than a keyword check because a commitment can be phrased in endless ways: "we'll ship that next month" contains no forbidden word. With these signals, there is a good setup to continuously improve the agent: bad conversations get surfaced, get improved, and become dataset items, to test against structurally. The team can also use this setup to safely try out a newer/cheaper/faster model, and make an informed decision on whether to deploy it or not. ## Conclusion Customer-facing automations often don't launch because the risk is high. This example outlines best practices for making it happen in a pragmatic way, embracing continuous learning and improvement. Check out the [other examples](/academy/examples) or the [academy](/academy) to learn more. --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/academy/examples/customer-support-chatbot.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .