--- title: Incident Response description: "How Langfuse handles incidents: the on-call rotation, the incident.io workflow, severity levels, and how status updates reach customers." --- # Langfuse Incident Response Plan ### Declaring an incident **Any Langfuse or ClickHouse team member** can declare an incident at any time — don't wait for certainty. When in doubt, page. Langfuse incidents are handled in the **ClickHouse incident.io account**. 1. Use `/inc` in Slack or declare the incident from incident.io. 2. Make it clear in the title and description that this is a **Langfuse** incident. 3. Include the affected service(s), region, customer impact, and useful links to errors, dashboards, traces, or support tickets. 4. Select the highest plausible severity — this is not published externally. For urgent issues, use `/inc escalate` to page Langfuse directly. You can also tag `@langfuse-oncall` in Slack. ClickHouse teammates can find the team in `#langfuse`; if the issue is related to ClickHouse instances, also involve `#ch-sre-team`. incident.io will automatically create a Slack channel and route the escalation to the Langfuse on-call team. Salesforce support escalations should use the same Langfuse on-call flow. See [On-call at Langfuse](/handbook/product-engineering/how-we-work/on-call) for how the rotation is staffed and reviewed. **When to page:** platform outages, ClickHouse instance issues affecting Langfuse, security issues, elevated errors, a broken core product flow, or a customer seeing another customer's data. ### Response The **first engineer to join** the incident channel is the **Incident Lead**. Assign yourself the role in incident.io. Pull in [DRIs](https://langfuse.com/handbook/how-we-work/ownership) of affected components, ClickHouse SREs, or Max if needed. For customer-facing incidents, pull someone from the business side to monitor Slack channels and support tickets. 1. **Triage** — Collect evidence (screenshots, metrics, logs), publish a status page update (see below). For critical incidents, enable the product announcement banner. 2. **Mitigate** — Restore the system first: rollback, scale up, feature-flag, hotfix. Root cause comes later. 3. **Stabilize** — Mark as mitigated in incident.io, update the status page, monitor for 15–30 min, then dissolve the call. ### War room call Keep all incident communication in the incident.io war room call so remote teammates can join quickly and we have a transcript for the post-mortem. 1. Open the incident Slack channel created by incident.io. 2. Click **☎️ Join the call** in the incident.io message. ### Status page Status pages are extremely important — they are our mechanism to show transparency to users, which builds trust. **When in doubt, always set up a status page.** The following should **always** have a status page: - Eval execution delays - Ingestion delays - Errors/latencies on public APIs - Login issues To publish: go to **"Status Pages"** in incident.io, select our public status page, and hit **"Publish Incident"**. Declaring an incident via `/inc` in Slack does **not** automatically create a public status page update — you must publish it separately. The incident lead keeps the [status page](https://status.langfuse.com) up to date with concise and accurate information throughout the incident. ### Post-mortem After mitigation, find and fix the root cause. Complete the post-mortem in Linear, in the incidents view, using the auto-generated timeline, covering: summary, impact, root cause, contributing factors, and action items with owners. Track follow-ups in the Linear ticket. Share in **#lf-team-engineering** in the ClickHouse Slack. --- ## Agent Instructions This page is part of the [Langfuse](https://langfuse.com) documentation, published as plain Markdown for AI agents. Every page is available as Markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` header. This page: `https://langfuse.com/handbook/product-engineering/incident-response.md`. ### Querying these docs If the answer is not on this page, query the documentation instead of guessing: - **Semantic search** across all Langfuse docs, returning an answer with the relevant pages and excerpts. Ask a specific, self-contained question: ```bash curl -sG "https://langfuse.com/api/search-docs" --data-urlencode "query=How do I trace a LangGraph agent?" ``` - **Index of every page**: , with per-section indexes [llms-docs.txt](https://langfuse.com/llms-docs.txt), [llms-integrations.txt](https://langfuse.com/llms-integrations.txt), and [llms-self-hosting.txt](https://langfuse.com/llms-self-hosting.txt). ### Before writing Langfuse code - **Install the [Langfuse Agent Skill](https://langfuse.com/docs/api-and-data-platform/features/agent-skill).** It encodes Langfuse's own best practices for instrumentation, prompt management, and evaluation, and materially improves results. - **Read [What does a good trace look like?](https://langfuse.com/docs/observability/best-practices.md)** before instrumenting an application. - **Verify endpoints, parameters, and response fields** against the [API reference](https://api.reference.langfuse.com) instead of inferring them from code examples. - **Use the [Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)** (`npx langfuse-cli api `) to read or write traces, prompts, datasets, and scores from the terminal. Found an error in these docs? Please open an issue at .