--- title: "From a manual spreadsheet to 12x higher experimentation velocity: how Slite observes and evaluates its AI agents with Langfuse" seoTitle: "How Slite Evaluates Its AI Agents with Langfuse" date: August 24, 2026 description: Learn how Slite replaced a CEO-run spreadsheet with a Langfuse dataset and evaluation pipeline, going from quarterly experiments to confident changes once a week. ogImage: /images/customers/slite/ogImage.jpg tag: customer-story author: Annabell customerLogo: "/images/customers/slite/slite-light.png" customerLogoDark: "/images/customers/slite/slite-dark.png" customerQuote: "Having this data set, having this whole test suite that a developer can run on their machine and quickly get results, it's a game changer for us." quoteAuthor: "Charley David" quoteRole: "Principal Engineer" quoteCompany: "Slite" quoteAuthorImage: "/images/customers/slite/charley-david.jpg" showInCustomerIndex: true --- [Customer stories](/users) › Slite ## Summary [#summary] **Self-maintaining knowledge base** Slite builds an agent that proofreads, moves, and archives company docs on its own, without being asked. **Deep agent observability** Langfuse traces show what the agents are doing as those workflows get more complex. **12x experimentation velocity** A ~100-item Langfuse dataset, judged on expected behavior, replaced a hand-run spreadsheet and moved experiments from quarterly to weekly. **Quality as scope expands** Developers and a background review agent get the visibility to grow scope of the agent without breaking what already works. ## A knowledge base that maintains itself [Slite](https://slite.com/) started as a knowledge base for companies. Ask it a question, and it finds the answer somewhere in your team's docs. Over the past two years that grew into workflows and assistants wired into tools like Slack and Intercom, and now into something more ambitious: an agent that watches signals across a company's other tools and proposes keeping documents current, moved, or archived on its own, without a person asking it to. "We evolved towards a knowledge base constructed by AI, that can be used with AI to ask questions about your content and your knowledge spread in every possible tool," says Charley David, Principal Engineer at Slite. "The goal is to keep your knowledge up to date without having the burden of having to manually go and update your document constantly." Today that's opt-in, document by document. The next release, planned for after summer, extends it toward what Charley calls "the complete auto-pilot of the whole knowledge base": every document, not just the ones a user has flagged. The more autonomy Slite hands the agent, the more that can go wrong, and the more the team needs to see what the agents are doing. ## Non-deterministic doesn't mean unobservable Slite adopted Langfuse early because they quickly realized that LLM agents, unlike traditional code, are non-deterministic and difficult to debug. Without visibility into tool usage, retrieval, and reasoning, the team was essentially debugging blindly. [Tracing](/academy/tracing) gave them that insight, so they could evaluate changes and build a dataset grounded in real-world behavior. That need for visibility ran into a real constraint: Slite's customers store proprietary knowledge, and data residency mattered enormously. Early on, tracing covered only Slite's own internal usage on Langfuse Cloud. Customer traffic went untouched. "We are also very wary about where the data of our users is ending up. Data residency is a key concern for our customers, as we are dealing with all company internals," Charley says. "So with Langfuse, at first we were just sending some of our traces. Now we send all of them." That constraint is what eventually pushed Slite to [self-host](/self-hosting). Langfuse was quickly up and running in their own infrastructure, and alongside the knowledge base itself they were able to trace and evaluate sensitive customer data. ## From a CEO's spreadsheet to a real evaluation loop For a long time, quality control at Slite was a spreadsheet of questions maintained by the company's CEO, Christophe Pasquier. Before any meaningful change shipped, he'd duplicate the sheet, run every question in the current model and harness configuration, and grade the answers himself, by hand. It worked, but it didn't scale. As Charley puts it, for a long time nothing like a real test suite existed for agents at all: "it was the far west." The unlock was rethinking what "the right answer" even means for a non-deterministic system. Instead of storing one fixed answer per question in a Langfuse [dataset](/docs/evaluation/experiments/datasets), Slite defines the behavior an answer needs to exhibit, and grades against that with an [LLM-as-a-judge](/docs/evaluation/evaluation-methods/llm-as-a-judge). "We don't store the expected answer, we store what the answer should contain. If you ask for your VAT number it should have the VAT number, and then the model is better at grading it and we get a more reliable outcome," says Charley David. This now lives as a managed [dataset](/docs/evaluation/experiments/datasets) in Langfuse with roughly 100 questions, first assembled from Christophe's original spreadsheet, now growing from real user thumbs-up/thumbs-down feedback and manual reviews, running against every change to the agent in [CI](/docs/evaluation/experiments/experiments-ci-cd). The runs are [evaluated](/docs/evaluation/overview) through an LLM-as-a-judge that compares the specified criteria against the AI-generated output. This allowed the team to go from quarterly to weekly experiments. That's roughly a **12x increase** in how often Slite can afford to test a new idea, and, just as importantly, a shift from a gut-feel discussion to one grounded in a number everyone trusts. It's not a hard merge gate. With non-deterministic outputs, and score, latency, and cost all trading off against each other, Slite treats it as a fast, trustworthy signal rather than a red/green flag. But it's the thing that tells a developer, before shipping, whether a reworked engine still performs at least as well as what's already in production. Slite is closing the [AI Engineering loop](/academy/ai-engineering-loop) with this setup, and found an effective way to ship changes frequently. ## Watching the agent watch itself The most forward-looking piece of Slite's setup is a background "retrospective" agent that samples a percentage of production traces via the [Langfuse API](/docs/api-and-data-platform/overview), reads the full trace, and writes up what looked noteworthy, both good and bad. A developer then reviews those write-ups in aggregate, not trace by trace. This already allowed Charley and the team to find places where the agent went off, at a volume that would not have been possible to handle otherwise. ![Diagram of Slite's retrospective review pipeline: traces logged in Langfuse, a retrospective agent sampling via the API, and a developer reading an aggregated digest](/images/customers/slite/slite-retrospective-trace-review.png) At Slite's volume, roughly 10,000 questions a day, each generating somewhere between 5 and 25 observations, with some running many more tool calls than that, there's no realistic way to review that by hand. Langfuse's API-first approach, [CLI](/docs/api-and-data-platform/features/cli), [MCP server](/docs/api-and-data-platform/features/mcp-server), and [agent skill](/docs/api-and-data-platform/features/agent-skill) compounds the effect. "It's super useful to have this available, you could run a suite of tests and ask Claude, 'find where my suite of tests performed poorly, can you find out why,' and it goes into very tiny details," Charley says. "That's really great." ## Why Langfuse Discovery came through a developer on Charley's team, Jason, who had already used Langfuse's Python SDK successfully before joining. From there, Charley ranks Slite's requirements in this order: 1. **Data residency and self-hostability.** "A lot of tools we can't use because they are US-based... it's a no-go for us." 2. **Ease of navigating traces and datasets**, both through the UI, the API, and the MCP. "The goal is to gain time when we want to understand what is happening on the trace or on volumes of traces." 3. **One place for everything**: traces, datasets, and runs together, rather than scattered across tools. Asked for three words to describe Langfuse, Charley didn't hesitate: **"Faster development cycles."** ## What's next Slite's roadmap moves in stages: from per-document maintenance (where they are today), to proposing changes across an entire workspace, to eventually an agent trusted enough to act without per-edit approval, once its track record earns that trust. None of that is viable without the [evaluation loop](/academy/ai-engineering-loop) scaling alongside it. Sustaining quality while scope keeps expanding is key. With new models shipping monthly and the agent's footprint only growing, Charley expects Slite's Langfuse value to keep increasing right alongside it: more traces stored, more retrospectives run, and more datasets to constantly evaluate whatever comes next. ## Ready to get started with Langfuse? Join thousands of teams building better LLM applications with Langfuse's open-source observability platform. - [Start free](/cloud) - [Documentation](/docs) - [Talk to an expert](/talk-to-us) --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/users/slite.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .