Latency percentile

Line every request up by how long it took, slowest last, then read off the value at a given position. p50 (the median) is the typical experience, p95 is the point where only the worst 5 percent are slower, p99 is the worst 1 percent. Averages hide exactly the requests that make people leave: 99 fast responses and one 30 second timeout average out to something that looks fine on a chart and feels terrible in a browser.

Reach for percentiles whenever you set a performance goal or an alert, because they are stated in units a human recognizes ('95 percent of checkout requests finish under 800ms'). Chart p50 and p95 or p99 together: p50 rising means everything got slower, p50 flat with p99 climbing means a subset is suffering, usually cold caches, a lock, a noisy neighbor, or one unlucky shard. p99 is also less rare than it sounds. If one page fans out to ten backend calls, most page loads touch somebody's p99, which is why tail latency dominates real-world speed.

Gotcha: you cannot average percentiles. Averaging the p99 of four servers, or of four one-minute buckets, produces a number that is not the p99 of anything. Percentiles have to be computed from the underlying distribution, which is why metric systems store latency as histogram buckets and estimate the percentile at query time. That estimate is only as good as your bucket boundaries, so pick buckets that straddle the range you actually care about.