How Trade Republic migrates live LLM workflows to agentic harnesses with Langfuse
Trade Republic's Ops Tools team on self-hosted Langfuse, and how homework and exam datasets make migrating production LLM processes to ReAct agents safer.
Highlights from the talk
Trade Republic is one of Europe's largest fintechs and a fully licensed bank, with over 10 million customers and €150 billion in assets under management. In this talk, Paolo Tamagnini (Senior Data Scientist, Ops Tools) explains how the team uses self-hosted Langfuse to migrate production LLM processes - sequenced prompts with predetermined flows - into more modern ReAct agent architectures with tools.
They started with Langfuse about two years ago on Postgres, then upgraded to ClickHouse when they moved to a newer version, and rolled it out from a single team to more teams while clearing bank compliance requirements.
"We're still a bank, so we had to follow very strict compliance - and we were able to go through it. We're using the open-source self-hosted version of Langfuse and we don't see any limitations there.
Many of their early "LLM processes" were easy for product teams to sketch in Figma and inspect in Langfuse: summarize an incident, classify it, score severity, trigger an action. That pattern worked well with earlier models. As reasoning models improved, they wanted agentic setups where a single prompt calls tools instead of a fixed chain of isolated prompts - but the workflows were already in production, so a naive rewrite was too risky.
Their answer is Langfuse plus the evaluation flywheel: trace every run, score good and bad examples (human annotation, deterministic checks, or LLM-as-a-Judge), fix failures (including via coding agents connected through the Langfuse MCP or CLI), and store cases in datasets.
"You can connect Claude Code to your Langfuse MCP - or the CLI directly - and it starts fixing and trying until it is fixed.
For architecture migrations they split data into a homework set (for iterating prompts and tools) and an exam set (held-out topics so coding agents cannot few-shot memorize the fix). An LLM judge compares the deployed workflow against the ReAct agent; only when both sets pass do they ship.
"You start with the homework set… you have your already deployed workflow and your ReAct agent, you make them compete, and a custom LLM judge scores the trace. Then you need the exam dataset to see if the issue was actually fixed - because coding agents are smart and can simply few-shot the homework.
Ready to get started with Langfuse?
Join thousands of teams building better LLM applications with Langfuse's open-source observability platform.
No credit card required · Free tier available · Self-hosting option