Peeking problem
The false-winner problem caused by checking an ordinary test repeatedly and stopping the first time the result turns significant.
See it
What it is
The peeking problem appears when you run an ordinary fixed-sample test, check it repeatedly, and stop the first time p drops below 0.05. Each look gives random noise another chance to cross the finish line, so the actual false-positive rate becomes larger than the alpha printed by the test. The p-value is valid for the sampling plan it assumed, not for 'keep looking until something exciting happens'.
The simple fix is a fixed horizon: calculate the sample size, cover the planned calendar cycles, and make the decision once. If early stopping matters, choose a sequential design before launch. Group-sequential methods such as O'Brien-Fleming use stricter boundaries at early looks and spend the error budget deliberately across the schedule. Optimizely rebuilt its statistics around this in 2015 with Stats Engine, precisely because customers watched the dashboard all day.
Gotcha: hiding the dashboard does not fix a plan that changes after someone sees the numbers. Dropping a losing variant, changing the primary metric, or extending only near-winners are all result-driven stopping decisions. Operational monitoring for crashes is fine; the problem is making efficacy calls with a method that did not account for those calls.
Ask AI for it
Replace this continuously watched fixed-horizon test with a group-sequential design. In R, use gsDesign::gsDesign with k = 5 planned looks, test.type = 4, alpha = 0.025 one-sided (equivalent to two-sided 0.05), beta = 0.20, sfu = sfLDOF for the O'Brien-Fleming efficacy spending function, and sfl = sfHSD for a non-binding futility bound. Print the cumulative sample size and both boundaries for every look, lock analysis to those five information fractions, and report continue, stop for efficacy, or stop for futility instead of an ordinary p < 0.05 rule at each look. Record the design before reading outcome data and add a warning if an analyst requests an unplanned look.