Summary

A knowledge base that maintains itself

Slite started as a knowledge base for companies. Ask it a question, and it finds the answer somewhere in your team's docs. Over the past two years that grew into workflows and assistants wired into tools like Slack and Intercom, and now into something more ambitious: an agent that watches signals across a company's other tools and proposes keeping documents current, moved, or archived on its own, without a person asking it to.

"We evolved towards a knowledge base constructed by AI, that can be used with AI to ask questions about your content and your knowledge spread in every possible tool," says Charley David, Principal Engineer at Slite. "The goal is to keep your knowledge up to date without having the burden of having to manually go and update your document constantly."

Today that's opt-in, document by document. The next release, planned for after summer, extends it toward what Charley calls "the complete auto-pilot of the whole knowledge base": every document, not just the ones a user has flagged.

The more autonomy Slite hands the agent, the more that can go wrong, and the more the team needs to see what the agents are doing.

Non-deterministic doesn't mean unobservable

Slite adopted Langfuse early because they quickly realized that LLM agents, unlike traditional code, are non-deterministic and difficult to debug. Without visibility into tool usage, retrieval, and reasoning, the team was essentially debugging blindly. Tracing gave them that insight, so they could evaluate changes and build a dataset grounded in real-world behavior.

That need for visibility ran into a real constraint: Slite's customers store proprietary knowledge, and data residency mattered enormously. Early on, tracing covered only Slite's own internal usage on Langfuse Cloud. Customer traffic went untouched.

"We are also very wary about where the data of our users is ending up. Data residency is a key concern for our customers, as we are dealing with all company internals," Charley says. "So with Langfuse, at first we were just sending some of our traces. Now we send all of them."

That constraint is what eventually pushed Slite to self-host. Langfuse was quickly up and running in their own infrastructure, and alongside the knowledge base itself they were able to trace and evaluate sensitive customer data.

From a CEO's spreadsheet to a real evaluation loop

For a long time, quality control at Slite was a spreadsheet of questions maintained by the company's CEO, Christophe Pasquier. Before any meaningful change shipped, he'd duplicate the sheet, run every question in the current model and harness configuration, and grade the answers himself, by hand. It worked, but it didn't scale. As Charley puts it, for a long time nothing like a real test suite existed for agents at all: "it was the far west."

The unlock was rethinking what "the right answer" even means for a non-deterministic system. Instead of storing one fixed answer per question in a Langfuse dataset, Slite defines the behavior an answer needs to exhibit, and grades against that with an LLM-as-a-judge. "We don't store the expected answer, we store what the answer should contain. If you ask for your VAT number it should have the VAT number, and then the model is better at grading it and we get a more reliable outcome," says Charley David.

This now lives as a managed dataset in Langfuse with roughly 100 questions, first assembled from Christophe's original spreadsheet, now growing from real user thumbs-up/thumbs-down feedback and manual reviews, running against every change to the agent in CI. The runs are evaluated through an LLM-as-a-judge that compares the specified criteria against the AI-generated output. This allowed the team to go from quarterly to weekly experiments.

That's roughly a 12x increase in how often Slite can afford to test a new idea, and, just as importantly, a shift from a gut-feel discussion to one grounded in a number everyone trusts.

It's not a hard merge gate. With non-deterministic outputs, and score, latency, and cost all trading off against each other, Slite treats it as a fast, trustworthy signal rather than a red/green flag. But it's the thing that tells a developer, before shipping, whether a reworked engine still performs at least as well as what's already in production.

Slite is closing the AI Engineering loop with this setup, and found an effective way to ship changes frequently.

Watching the agent watch itself

The most forward-looking piece of Slite's setup is a background "retrospective" agent that samples a percentage of production traces via the Langfuse API, reads the full trace, and writes up what looked noteworthy, both good and bad. A developer then reviews those write-ups in aggregate, not trace by trace. This already allowed Charley and the team to find places where the agent went off, at a volume that would not have been possible to handle otherwise.

At Slite's volume, roughly 10,000 questions a day, each generating somewhere between 5 and 25 observations, with some running many more tool calls than that, there's no realistic way to review that by hand.

Langfuse's API-first approach, CLI, MCP server, and agent skill compounds the effect. "It's super useful to have this available, you could run a suite of tests and ask Claude, 'find where my suite of tests performed poorly, can you find out why,' and it goes into very tiny details," Charley says. "That's really great."

Why Langfuse

Discovery came through a developer on Charley's team, Jason, who had already used Langfuse's Python SDK successfully before joining. From there, Charley ranks Slite's requirements in this order:

1. Data residency and self-hostability. "A lot of tools we can't use because they are US-based... it's a no-go for us."
2. Ease of navigating traces and datasets, both through the UI, the API, and the MCP. "The goal is to gain time when we want to understand what is happening on the trace or on volumes of traces."
3. One place for everything: traces, datasets, and runs together, rather than scattered across tools.

Asked for three words to describe Langfuse, Charley didn't hesitate: "Faster development cycles."

What's next

Slite's roadmap moves in stages: from per-document maintenance (where they are today), to proposing changes across an entire workspace, to eventually an agent trusted enough to act without per-edit approval, once its track record earns that trust.

None of that is viable without the evaluation loop scaling alongside it. Sustaining quality while scope keeps expanding is key. With new models shipping monthly and the agent's footprint only growing, Charley expects Slite's Langfuse value to keep increasing right alongside it: more traces stored, more retrospectives run, and more datasets to constantly evaluate whatever comes next.