Langfuse Incident Response Plan

Langfuse escalations and incidents are handled in the ClickHouse incident.io account.

Escalating to on-call

Any Langfuse or ClickHouse team member can escalate to the Langfuse on-call engineer at any time — don't wait for certainty. When in doubt, page.

1. Use /incident escalate (short: /inc esc) in Slack and select Langfuse as the escalation path to page the current on-call engineer.
2. In the notification message, describe what you are seeing: affected service(s), region, customer impact, and useful links to errors, dashboards, traces, or support tickets.
3. Set the priority and click Escalate.

For example, to report that the scores view is down, select Langfuse as the escalation path and use a notification message like "Scores UI view is no longer responding across multiple projects":

Escalate instead of declaring an incident yourself. The on-call engineer evaluates the issue and decides whether it needs a formal incident (see below).

You can also tag @langfuse-oncall in Slack. ClickHouse teammates can find the team in #langfuse; if the issue is related to ClickHouse instances, also involve #ch-sre-team. Salesforce support escalations should use the same Langfuse on-call flow. See On-call at Langfuse for how the rotation is staffed and reviewed.

When to escalate: platform outages, ClickHouse instance issues affecting Langfuse, security issues, elevated errors, a broken core product flow, or a customer seeing another customer's data.

Declaring an incident

The on-call engineer decides whether an escalation becomes an incident. Declare an incident when the response needs coordination between multiple people or teams, for example:

• the issue is severe and needs to be documented with a post-mortem,
• other teams at ClickHouse need to be involved, or
• stakeholders outside engineering need to be kept informed.

Issues the on-call engineer can resolve on their own don't need an incident. Status page updates don't require an incident either; they can be published independently.

To declare an incident:

1. Use /incident (short: /inc) in Slack or declare the incident from incident.io.
2. Make it clear in the title and description that this is a Langfuse incident.
3. Include the affected service(s), region, customer impact, and useful links to errors, dashboards, traces, or support tickets.
4. Select the highest plausible severity — this is not published externally.

incident.io will automatically create a Slack channel for the incident.

Response

The first engineer to join the incident channel is the Incident Lead. Assign yourself the role in incident.io. Pull in DRIs of affected components, ClickHouse SREs, or Max if needed. For customer-facing incidents, pull someone from the business side to monitor Slack channels and support tickets.

1. Triage — Collect evidence (screenshots, metrics, logs), publish a status page update (see below). For critical incidents, enable the product announcement banner.
2. Mitigate — Restore the system first: rollback, scale up, feature-flag, hotfix. Root cause comes later.
3. Stabilize — Mark as mitigated in incident.io, update the status page, monitor for 15–30 min, then dissolve the call.

War room call

Keep all incident communication in the incident.io war room call so remote teammates can join quickly and we have a transcript for the post-mortem.

1. Open the incident Slack channel created by incident.io.
2. Click ☎️ Join the call in the incident.io message.

Status page

Status pages are extremely important — they are our mechanism to show transparency to users, which builds trust. When in doubt, always set up a status page.

The following should always have a status page:

• Eval execution delays
• Ingestion delays
• Errors/latencies on public APIs
• Login issues

To publish: go to "Status Pages" in incident.io, select our public status page, and hit "Publish Incident". Declaring an incident via /inc in Slack does not automatically create a public status page update — you must publish it separately.

The incident lead keeps the status page up to date with concise and accurate information throughout the incident.

Post-mortem

After mitigation, find and fix the root cause. Complete the post-mortem in Linear, in the incidents view, using the auto-generated timeline, covering: summary, impact, root cause, contributing factors, and action items with owners. Track follow-ups in the Linear ticket. Share in #lf-team-engineering in the ClickHouse Slack.

---

---