Guardrail metric
The metric you watch so a win does not hide a loss: signups went up, but did refunds, load time, or support tickets go up with them?
See it
What it is
Your primary metric is the one you are trying to move. A guardrail is the one that must not move the wrong way while you do it. Push signups with an aggressive modal and watch refund rate, support tickets, unsubscribe rate, page load time and crash-free sessions. Mature experiment programs (Microsoft's ExP, Airbnb, Booking) run every test against a standing guardrail set plus one or two picked for that specific change.
Declare them before launch, with numbers: the current baseline, the worst degradation you would accept, and what happens if it is crossed (ship anyway, hold, roll back). Deciding after the results are in is how a 3 percent conversion lift ships alongside a 12 percent jump in refunds.
Gotcha: guardrails invert the statistics. 'Not statistically significant' does not mean 'safe', it usually means underpowered, and a wide confidence interval can easily hide real damage. Check the interval against your tolerance instead of reading a p-value, and mind which way the metric points: for something you want high (crash-free sessions, retention) the interval has to stay above your floor, and for something you want low (refunds, tickets, p95 latency) it has to stay below your ceiling. That is a non-inferiority test, and getting the direction backwards passes exactly the metrics that should have failed. Keep the set small too: every extra guardrail buys another false alarm.
Ask AI for it
For this experiment [describe the change and the primary metric], define the guardrail set before launch. Give me 3 to 5 guardrails covering user harm, revenue quality, performance and support load. For each one: the baseline, taken from the numbers I supplied or from a query you run against my data (write the query out); if neither is available, mark the baseline TBD and say what to query, and never invent a plausible-looking figure. State each metric's direction, meaning whether up is good or bad, and set the non-inferiority bound to match: for a benefit metric the confidence interval on the change must stay above a lower bound, for a harm metric like refunds, tickets, crashes or latency it must stay below an upper bound. Also give how it is measured, and the action if the bound is crossed (ship, hold, roll back). Frame every check as non-inferiority against that bound, not a p-value test, and run a power calculation per guardrail showing the sample size needed to detect damage at its threshold, flagging any that our traffic cannot support.