Cardinality explosion
When metric labels combine into millions of separate series, making storage, queries, and the monitoring bill blow up.
See it
What it is
A cardinality explosion happens when a metric's labels combine into far more distinct time series than expected. Ten methods, 100 routes, 20 regions, and 50 status values have room for one million combinations. Add user_id or a raw path and the ceiling stops being practical. Each series needs index and memory overhead even if it receives only a few points.
Look for it when a metrics bill jumps, ingestion slows, queries time out, or Prometheus memory climbs after an instrumentation change. Keep metric labels bounded: route templates instead of raw URLs, status classes instead of error messages, and small enumerations instead of identifiers. Put per-user or per-request detail in traces and structured logs.
Gotcha: multiplying the documented value counts gives a worst-case ceiling, not the number of series that actually exist. Measure active series in the backend and rank metrics by series count before deleting useful labels. The sneakiest offenders are generated values such as pod names, UUIDs, timestamps embedded in errors, and URLs with IDs.
Ask AI for it
Audit this Prometheus instrumentation for a cardinality explosion. List active series by metric name, rank label keys by distinct value count, and identify raw URLs, user_id, session_id, UUIDs, pod names, and error messages used as labels. Replace paths with route templates, collapse HTTP statuses to 2xx, 4xx, and 5xx where exact codes are unnecessary, and move request-specific values to OpenTelemetry span attributes and structured logs. Add a prometheus_tsdb_head_series alert and a CI test that rejects metric label keys outside an explicit allow-list.