Alert threshold

A threshold turns a chart into a decision. You pick a number and a duration ('5xx rate above 2 percent for 5 minutes'), and when the metric crosses it the rule fires a notification: Slack, email, or a page at 3am. The duration part matters as much as the number. Without it, one noisy scrape flips the alert on and off all night, which is called flapping.

Good thresholds sit on symptoms users can feel (error rate, latency, checkout failures), not on causes that may be harmless (CPU at 90 percent is fine if nobody is waiting). Set them from your own baseline: look at a normal week, then place the line above the usual peak but below the level where customers notice. Two tiers help, a warning that goes to a channel and a critical that pages. When a fixed number cannot capture normal, the alternatives are anomaly detection (deviation from learned behavior) and burn-rate alerts (how fast you are eating an SLO's error budget), which fire on trajectory rather than a static line.

Gotcha: the failure mode is not a missed alert, it is alert fatigue. Every threshold that fires without needing a human trains the on-call to ignore the next one. If an alert has no action attached, it should be a dashboard, not a page. Also watch percentage rules on tiny traffic: 1 failed request out of 3 at 4am is a 33 percent error rate, so gate the rule on a minimum request count.