Change failure rate
The share of production changes that cause trouble serious enough to need a rollback, hotfix, fix-forward, or incident response.
What it is
Change failure rate is the percentage of production changes that require remediation because they degraded the service. The numerator includes changes followed by a rollback, fix-forward, hotfix, or incident; the denominator is all production changes in the same period. Ten failed changes among 200 gives a 5% rate.
Reach for it when release speed alone is hiding the cost of unstable delivery. Track it per service and as a trend, then inspect the failures for repeated causes such as weak tests, risky migrations, or missing canary checks.
Gotcha: the number is easy to distort. Counting pipeline retries as production changes bloats the denominator, while counting only formal incidents misses quiet rollbacks and emergency fixes. Write down the failure window and remediation events, and keep the rule stable before using the trend.
Ask AI for it
Calculate change failure rate from the GitHub Deployments API and PagerDuty incidents. Treat each successful deployment to the production environment as one change, deduplicate by immutable artifact digest, and mark it failed when a linked PagerDuty incident, rollback deployment, or hotfix occurs within 24 hours for the same service. Store the commit SHA, service, deployment ID, failure reason, remediation type, and timestamps. Report failed production changes divided by all production changes as a percentage for rolling 7-day and 30-day windows, list unlinked remediation events for review, and add tests for redeploys, batched commits, and a failure that crosses a window boundary.