Runbook
The step-by-step document linked from an alert, telling the on-call person how to diagnose, mitigate, verify, and escalate the failure.
See it
What it is
A runbook is the step-by-step document attached to a specific alert or failure: how to confirm the problem, which dashboards and queries to open, safe mitigation steps, how to verify recovery, and when to escalate. It turns knowledge trapped in one engineer's head into instructions a tired responder can follow at 3am.
Write one when an alert first becomes pageable, then improve it after every incident. Start with the exact alert name and user impact, include copyable read-only diagnostic commands, put the safest reversible mitigation before the clever diagnosis, and link directly to the right dashboard view. Prometheus alert annotations commonly carry a runbook_url so the page opens at the instructions.
Gotcha: stale runbooks are confidently wrong, which is worse than no runbook. Keep them beside the code when possible, assign an owner and review date, and test commands without waiting for an outage. Never paste secrets into the page or make a destructive command the first step. If a responder must improvise each time, feed the missing step back into the document.
Ask AI for it
Create a Markdown runbook for every paging Prometheus alert in this repository. Use the alert name as the title and include user impact, prerequisites, direct Grafana links, copyable read-only diagnostic queries, the safest reversible mitigation, recovery verification, rollback, escalation criteria, owner, and review date. Add a runbook_url annotation to each Prometheus rule that points to its document. Put dangerous commands behind an explicit warning and a confirmation step, use placeholders instead of secrets, and add a CI check that fails when a paging rule has no reachable runbook.