How Trade Republic migrates live LLM workflows to agentic harnesses with Langfuse
Trade Republic's Ops Tools team on self-hosted Langfuse, and how homework and exam datasets make migrating production LLM processes to ReAct agents safer.
Highlights from the talk
Trade Republic is one of Europe's largest fintechs and a fully licensed bank, with over 10 million customers and €150 billion in assets under management. In this talk, Paolo Tamagnini (Senior Data Scientist, Ops Tools) explains how the team uses self-hosted Langfuse to migrate production LLM processes - sequenced prompts with predetermined flows - into more modern ReAct agent architectures with tools.
They started with Langfuse about two years ago on Postgres, then upgraded to ClickHouse when they moved to a newer version, and rolled it out from a single team to more teams while clearing bank compliance requirements.
"We're using the open-source self-hosted version of Langfuse and we don't see any limitations there. We're super happy with it.
Many of their early "LLM processes" were easy for product teams to sketch in Figma and inspect in Langfuse: summarize an incident, classify it, score severity, trigger an action. That pattern worked well with earlier models. As reasoning models improved, they wanted agentic setups where a single prompt calls tools instead of a fixed chain of isolated prompts - but the workflows were already in production, so a naive rewrite was too risky.
Their answer is Langfuse plus the evaluation flywheel: trace every run, score good and bad examples (human annotation, deterministic checks, or LLM-as-a-Judge), fix failures (including via coding agents connected through the Langfuse MCP or CLI), and store cases in datasets. For architecture migrations they split data into a homework set (for iterating prompts and tools) and an exam set (held-out topics so coding agents cannot few-shot memorize the fix). An LLM judge compares the deployed workflow against the ReAct agent; only when both sets pass do they ship.
"You start with the homework set… you have your already deployed workflow and your ReAct agent, you make them compete, and a custom LLM judge scores the trace. Then you need the exam dataset to see if the issue was actually fixed - because coding agents are smart and can simply few-shot the homework.
Ready to get started with Langfuse?
Join thousands of teams building better LLM applications with Langfuse's open-source observability platform.
No credit card required · Free tier available · Self-hosting option