Alert fatigue
When so many alerts are noisy or pointless that responders start ignoring them, including the one that matters.
See it
What it is
Alert fatigue is what happens when responders see so many noisy, duplicate, or unactionable alerts that they stop treating any one of them as urgent. Every page that needs no human action teaches the on-call person that the next page can probably wait too. The dangerous part is not the annoyance, it is the real outage hidden in the pile. The phrase is borrowed from hospital alarm fatigue, where nurses surrounded by constantly beeping monitors stop hearing the one that means something.
Reach for alert cleanup when pages fire during normal operation, several rules announce the same incident, or the first response is always to silence the alert. Rob Ewaschuk's 'My Philosophy on Alerting', written at Google and reprinted in the SRE book, gives the rule most teams eventually land on: page only for urgent user-visible symptoms that need a person now. Send warnings to a ticket or dashboard, group related notifications, and attach a runbook so the action is obvious. Prometheus burn-rate alerts are useful here because they page on how quickly an SLO budget is disappearing, not every brief wobble.
Gotcha: muting noisy rules without finding out why they fire can trade fatigue for blind spots. Review alert history, record whether each page was actionable, then delete, retune, or reroute it with evidence. Keep an eye on alert volume per shift after cleanup, because noise creeps back one well-meant rule at a time.
Ask AI for it
Audit this project's Prometheus alerting rules and Alertmanager config for alert fatigue. For every rule, identify the user-visible symptom, the required human action, and the runbook URL; turn anything without an urgent action into a ticket or dashboard instead of a page. In Alertmanager, group notifications by service and alertname, add inhibition rules so a service-down alert suppresses its downstream symptoms, and set explicit group_wait, group_interval, and repeat_interval values. Replace static symptom thresholds with multi-window SLO burn-rate alerts where an SLO exists, and produce a before-and-after table of expected pages per week.