In regulated industries, you cannot ship an AI agent and fix it in production. One wrong answer can be a compliance incident. But a human checking every case defeats the point of having an agent.

We will show you how teams in regulated industries get there, from the first prototype to an agent that handles part of the work on its own.

What makes regulated industries different

AI development teams in regulated industries usually face three requirements for their projects.

Evidence before launch. Banks already run risk processes that ask for test results, review sign-offs, and documentation before an application reaches production (deployment gates for LLMs in financial services). The review board will want to see how the agent behaves in realistic cases.

Oversight after launch. Regulators expect companies to stay in control. FINRA's 2026 oversight report asks firms to define where humans review agent actions, how they track agent decisions, and how agent behavior is checked for compliance. Article 14 of the EU AI Act requires that people can effectively oversee high-risk AI systems.

Control over data. Conversations contain account details, transactions, and health information. Teams need to know where that data is stored, what gets redacted, and who can see it. In healthcare, a business associate agreement has to be in place before any protected health information is processed.

The approach: start the loop offline

The AI engineering loop describes how teams improve AI applications: trace, monitor, build datasets, run experiments, and evaluate, and then repeat.

Many teams enter the loop in production, with tracing and monitoring live traffic. Regulated teams run most of it before going to production. They trace offline runs, build datasets, run experiments, and evaluate until the agent meets the quality bar. Only then do they monitor live traffic. The eight steps below show how.

1. Set up tracing

Add tracing before anyone outside the team uses the agent. Traces provide you insight into what happened during a run and provide you with a way of showing evidence of what the agent did. Record the prompt version, model, tool calls, retrieved context, and any human approval on every run. Decide where to mask sensitive fields before they are stored. The SDKs support client-side masking, so account numbers or patient identifiers can be redacted at the source.

2. Run example inputs through the agent

Next, run a set of examples through the agent that roughly matches what you expect to hit your system in production. Each run produces a trace you can learn from. At this point, you do not need expected answers yet. Good sources for cases are:

• Synthetic examples. Generate inputs that cover typical requests and their variations. Domain experts can add cases they know from experience.
• Historical data. Run the agent on cases your team has already decided, such as resolved support tickets or reviewed claims.
• Shadow runs. Let the agent process live cases next to the current workflow without acting on anything. This gives you traces from live data without bringing the agent into customer contact.

3. Find failure modes with error analysis

Now have your domain experts read those traces. For each one, they note the first thing that went wrong, then group the notes into failure modes. This process is called error analysis.

Some failures show that the agent needs more specification, such as a missing rule in the prompt, and you can fix them right away. Others keep coming back and need systematic evaluation.

4. Define what to evaluate, including compliance

What you evaluate comes from two places:

• Failure modes. The recurring failures from error analysis become quality evaluators. Generic metrics such as helpfulness rarely match how your agent fails, and they can skew your view on quality.
• Compliance requirements. Agree with compliance on what the agent must never do. For a bank, that could be giving investment advice, quoting a rate that is not on the price list, or sending a customer message without approval. Turn them into evaluators.

A code check, such as a regex for a missing disclaimer, is the preferred choice. Use LLM-as-a-judge for anything that needs judgment, such as a response that sounds like investment advice.

5. Build a ground truth dataset

With the evaluation criteria in place, build a ground truth dataset together with your domain experts. It should contain your typical inputs, the cases that could go wrong, and cases that should trigger each compliance limit, along with how the agent should handle each one.

Experts disagree more often than teams expect. To get to a reliable ground truth dataset, let your experts review each other's work. Ideally, bring the experts into one room to talk through those cases, agree on the right handling, and write the missing rule into your guidelines.

In Langfuse, annotation queues let several experts label the same items. Score analytics then calculates percent agreement and Cohen's kappa between them, so you can see which cases need a discussion.

6. Iterate until the agent meets the quality bar

Run the dataset as an experiment and record the scores as your baseline. Then review failed cases and improve the agent by changing its context, tool calls, or prompts, rerun the experiment, and compare it with the previous run.

7. Set up production monitors

Once you are happy with the results, decide which evaluators should also run in production with online evaluation. Compliance evaluators belong on every trace. Add the quality evaluators you want to keep watching live, for example whether answers are phrased the way your guidelines require.

For chat agents, also track explicit feedback and signs of trouble, such as a user asking for a person or correcting the agent.

8. Go live and run the AI engineering loop

Now go live, depending on the risk profile with a partial or complete human in the loop setup. From here on, you are in the regular AI engineering loop: trace production, watch the monitors, turn interesting traces into new dataset items, and rerun the dataset before every release.

Keep humans on a subset, or on everything at first. At launch, let the agent escalate a little too often to a human. A reply like "I can't help with that, I'll connect you to a person" costs a few minutes of staff time, and a wrong answer in a regulated setting can cost much more. The amount of human review shrinks as the evidence grows. A common pattern, described by Coinbase at AI Engineer NYC, has three steps:

1. Supervised. Reviewers check every output.
2. Sampled. Once measured quality passes a threshold set by policy, reviewers check a sample.
3. Exceptions. Reviewers mainly handle flagged cases and keep improving the datasets.

Match the review rate to the risk. Guidance from the Joint Commission and CHAI recommends checking tools that inform clinical decisions more often than tools that help with administrative tasks. The same logic applies in a bank: a credit decision deserves more review than a branch-hours question.

Automate the slices that work. Asking a person to approve everything does not scale, and reviewers who approve hundreds of routine cases a day stop looking closely. Classify incoming requests by category, let the agent handle categories that score well in your dataset and in production, such as a password reset, and keep sending the rest to a person.

How Langfuse supports each step

Conclusion

Regulated teams can bring AI agents to production. Most of the work happens before launch: trace from the start, learn how the agent fails, agree with compliance on what to evaluate, and test the agent against a ground truth dataset your experts trust. The same traces, datasets, and scores then become the evidence for your review board. After launch, keep people on the cases that need them and hand more categories to the agent as the evidence grows. Start with one workflow that has clear rules and a team that already reviews the work by hand, and expand from there.

Further reading

• Langfuse for financial services: data protection, deployment options, and compliance documentation for banks and insurers.
• Building deployment gates for LLMs and AI agents in financial services: how to turn datasets and evaluators into pass or fail checks that a review board can inspect.
• Customer stories from regulated industries: DKB, Trade Republic, and Merck.
• Example: Automate the parts that work.

---

---