Anomaly detection
Alerting when a signal strays from its learned normal pattern, so expected daily peaks do not need one rigid threshold.
See it
What it is
Anomaly detection flags values that depart from a learned baseline instead of crossing one fixed line. A checkout rate of 20 per minute might be normal at 3am and alarming at noon, so a useful model learns trend and seasonality, then scores how far the latest point sits outside its expected band. The named techniques underneath are older than the dashboards that sell them: Holt-Winters seasonal smoothing dates to 1960, and STL seasonal-trend decomposition to 1990.
Reach for it when normal changes by hour, weekday, or season and a fixed threshold either misses real drops or pages on every predictable peak. It works best on stable, high-volume signals with repeated patterns, such as requests, signups, and job throughput. Keep a simple hard limit for conditions that are always bad, such as disk space reaching zero.
Gotcha: unusual is not the same as harmful. A launch, holiday, deploy, or tracking bug can all look anomalous, while a slow drift may teach the model that broken is the new normal. Give alerts a minimum impact gate, retrain deliberately, and show the observed value, expected band, and business symptom so the on-call has something actionable.
Ask AI for it
Build an anomaly alert for the Prometheus request-rate series using Python, statsmodels MSTL, and a robust z-score based on median absolute deviation. Train on four complete weeks of five-minute samples, model daily and weekly seasonality, and score both upward spikes and downward drops. Fire only when three consecutive points exceed the band and the absolute change is at least 20 percent. Exclude missing scrapes and annotated deployment windows from training. POST to the Alertmanager /api/v2/alerts endpoint with the observed value, expected value, band, score, and a dashboard link, and backtest the rule against the previous week before enabling paging.