Evals for High-Stakes AI Agents

Some AI agents can afford a few mistakes. A customer-facing agent at a bank cannot. A wrong answer can carry financial, legal, and reputational risk, and the system can look healthy while it happens.

In this workshop from AI Engineer New York City, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop with a banking assistant. It is for teams in financial services, healthcare, and other regulated industries that have a working prototype and now need to answer leadership's question: is this agent safe to put in front of customers?

What you will learn

• How teams use the AI engineering loop to successfully trace and evaluate their applications
• How regulated teams build a ground truth dataset when they do not have one yet
• How to trace an agent with PII masking so sensitive data never lands in your observability tool
• How to build datasets from spreadsheets and from production traces
• How to run expert review with annotation queues and turn free-text comments into failure modes
• Why to combine LLM-as-a-judge and code evaluators
• How to run evals automatically on every prompt change and block it in CI when scores drop
• How to build dashboards for business stakeholders
• Where to use your coding agent, and where to keep humans in the loop

The AI engineering loop

The workshop starts with the AI engineering loop, the practices that teams who reliably ship and improve agents tend to follow:

• Tracing records every step your agent takes, in a form you can read yourself and your coding agent can work through at scale.
• Monitoring tracks cost and latency across traces, plus quality signals like whether compliance guardrails held or how many users are frustrated.
• Datasets are a representative set of cases your agent should handle, together with how it should handle them. That includes out-of-scope requests it should decline, like giving investment advice. The expected output can be a sample reply, an expected behavior, or keywords that must be covered.
• Experiments change one thing at a time (a prompt, a model, the context, a tool description), so you can tell what caused the result.
• Evaluation measures experiment results at scale with LLM-as-a-judge and code-based checks, while you still read some outputs yourself.

What you learn in production feeds back into your datasets, and the loop starts again.

Where regulated teams start

The right entry point depends on the stakes. A team building a low-stakes internal chat agent can start with tracing and a few monitors, go to production, and add test cases over time.

Customer-facing agents in regulated industries need more up front. Tracing is still the foundation: what steps the agent took, what inputs and outputs it saw, and which tools it called. On top of that, these teams need a ground truth dataset. There are three ways to build one:

• Expert curation. Work with domain experts in your company to build the dataset directly.
• Shadow runs. Keep running the process manually, run the agent in parallel, and compare how the agent would have handled each case. This measures accuracy and builds your ground truth at the same time.
• Human in the loop first. Run a fully human-reviewed system, then over time split the work into parts you are confident to automate and parts where a human always stays involved. Agents for medical device compliance are one place this path has been used.

Demo: a banking assistant with tracing and PII masking

The demo is a typical banking assistant (banking assistant demo repo) with a few features that matter for financial services:

• Persona impersonation. You can act as different kinds of customers, from an everyday account holder to an ultra high net worth individual, to check that the agent respects the guardrails for each.
• PII masking. When a user types their credit card number in plain text, the trace in Langfuse shows it fully redacted. A masking function removes anything defined as PII before it is persisted. Financial institutions want this kind of data to stay in their own systems of record.
• Scores on every trace. Each new trace is scored automatically: whether the answer is compliant with banking information, which failure modes occurred, and whether guardrails blocked anything.

Ask the assistant "Can I put my savings in Bitcoin?" and it declines. In the trace, the input guardrail names investment advice as the primary risk, and a categorical score records the customer's intent as investment advice, so these cases are easy to filter for later.

The demo also handles voice. A recorded contact center call goes through speech to text, the transcript is evaluated, and the trace links to the original audio along with the assistant's response. See multi-modality for how media is attached to traces.

Building datasets

Customers will use your system in ways you did not design for, and you cannot predict every output of an LLM. The evaluation work starts from datasets.

A dataset is a set of inputs and expected outputs. If you come from software development, these are your test cases. From the demo:

• The expected output does not have to be the exact answer. It can describe the intent or behavior you expect.
• Use several datasets for different areas of quality. The demo has one for red teaming (prompt injection and attempts to get around guardrails) and one for unresolved disputes.
• Two ways to create them. Many teams already have a spreadsheet of what they want to test, which you can upload as a CSV. You can also add production traces to a dataset once you decide they are good examples of what future outputs should look like.

Everything shown in the UI also works through the CLI, the MCP server, or the API.

Expert review and failure modes

Before you can measure quality, you need to know what goes wrong. Reading traces to check whether the agent took the right steps, called the right tools, and produced the right output is error analysis. That reading can take days.

You cannot read thousands of production traces by hand. Instead:

1. Sample. Curate a representative slice of production, say 10% or 20%, and filter for what matters, for example conversations about disputes or credit cards.
2. Send it to an expert. Put the sample into an annotation queue for a domain expert, such as your risk expert.
3. Keep it simple for them. Experts have a day job. Give them a couple of scores (yes or no, true or false, a 1 to 10 scale) and a comment field. In the demo, flagging a missing citation takes about ten seconds. A hundred traces at around 30 seconds each is under an hour of their time, and they can dictate their comments.
4. Let an LLM find the patterns. The annotations stay attached to the traces as evidence. An LLM then groups the raw comments into recurring failure modes. Experts give unfiltered feedback instead of inventing categories, and the LLM does the pattern matching.

Those failure modes become your quality dimensions, the scores you measure every experiment against. In the demo they include expected source correctness, formal register (addressing customers with the right level of formality and matching their language), must-include content, and no unsolicited upsell.

Choosing evaluators

Next, decide how to grade each failure mode. Evaluators can score traces automatically as they come in, and your application can also send scores in real time.

Use a mix of LLM-as-a-judge and deterministic evaluators, depending on the failure mode:

• LLM-as-a-judge handles judgment calls, but some of its grades will be wrong, so it should not be your only check.
• Code evaluators are plain functions. The demo uses a simple regex for an exact match. The same input always returns the same output.

Together they are more reliable than either one alone.

Running evals automatically on every prompt change

An experiment runs your agent (system prompt, model, tools) against a dataset, so you can test a change to one of them before promoting it to production.

The demo wires this into CI so nobody has to remember to run it:

1. Version your prompts. The system prompt lives in Langfuse prompt management with its full version history. Some customers keep a copy in GitHub or GitLab too.
2. Trigger on change. An automation fires on any prompt change and launches a CI job in GitHub. See the GitHub integration and webhooks.
3. Promote a version. In the demo, version 4 is promoted to the production label, which triggers the CI job. This works from the UI, the API, or the CLI.
4. Run the eval gate. The CI job runs the experiment against the dataset and returns pass or fail across all evaluators: must include, source recall, language match, correctness, and no unsolicited upsell. Each score can have a minimum threshold, and the change passes only if it clears all of them.

In the demo, the gate runs after production has already moved. If your application fetches the prompt by the production label, it serves the new version before the checks finish, and a failed job cannot stop that. In your own setup, put the new version on a candidate label such as staging, run the gate on that, and move production only after it passes.

The demo leaves the final merge to a human, though the pipeline could also open a PR for review. Either way, there is a record of why a configuration went to production. Learn more in experiments in CI/CD.

Dashboards for stakeholders

Business stakeholders want to see that the agent delivers value. The demo builds a custom dashboard from the data the agent already produces:

• Directional business value. About 1,400 interactions in the last seven days that did not need a human agent, at a total cost of under $7. The logic will differ by organization, but it shows a direction.
• Failure modes over time. Out of scope, informal register, and unresolved disputes are the top three and creeping up. Instead of searching hundreds of thousands of traces, filter tracing for unresolved disputes, save the filter as a view, and investigate a few dozen traces.
• User feedback. Thumbs up and thumbs down captured as scores. Most are positive, so focus on the 18% that are not.
• Security mix. Most interactions carry no risk, but 89 in the last seven days involved investment advice. Filter by the categorical security risk score to see those cases.

Working with your coding agent

Some teams hook their coding agent up to their traces and say "Please fix my agent. Make no mistakes." That mostly produces slop.

The chart plots time invested against system quality. Letting the agent fix everything on its own costs no time and gives you a low-quality system. Doing everything by hand is no longer necessary either. Aim for the balance in between:

• Automate instrumenting your app, finding failure modes, running evals, and running 20 experiments where you would have come up with one.
• Bring your taste and your domain experts to build and curate datasets, decide what problem the application should solve, and set its direction.

Over time, you can automate more of the loop.

If you start tomorrow

1. Install the Langfuse Agent Skill. It is a set of markdown files that teaches your coding agent how to work with Langfuse, connected to the docs, the integration pages, and the Academy best practices on datasets, instrumentation, and evaluators.
2. Look at data manually. Use the skill for what can be automated, and curate a set of traces to review yourself.
3. Collect ground truth. Use historical data, production traces, or samples run with your domain experts.
4. Define your hard limits. In a regulated space, write down what must never happen. For financial services, the agent should never give investment advice. Put those cases in your evaluation set so you know when it happens.

Chapters

• 00:00 Why evals are hard
• 05:01 The AI engineering loop
• 12:47 Demo: a banking assistant with tracing and PII masking
• 20:35 Building datasets
• 25:18 Expert review and failure modes
• 30:07 Choosing evaluators
• 35:00 Running evals automatically on every prompt change
• 39:44 Dashboards for stakeholders
• 45:58 Working with your coding agent

Resources

• Banking assistant demo repo
• Langfuse for financial services
• Langfuse Academy, the AI engineering loop, and error analysis
• Langfuse workshop: a self-paced walkthrough of every step of the loop
• Masking sensitive data
• Annotation queues
• LLM-as-a-judge and code evaluators
• Datasets and experiments in CI/CD
• Custom dashboards
• Langfuse Agent Skill

---

---