Experiment sample-size calculation
The pre-test calculation of how many users each variant needs to reliably detect the smallest change worth acting on.
See it
What it is
A sample-size calculation tells you how many experimental units each variant needs before the test starts. It combines the current baseline rate, the smallest effect worth detecting, the tolerated false-positive rate (alpha), the desired power, and the allocation between variants. Smaller effects hide more easily in noise, so detecting them requires far more users.
Reach for it before every experiment, then turn the answer into a run time using eligible traffic rather than total site traffic. Eighty percent power means that, across repeated tests where the true effect equals the planned minimum detectable effect, about four in five would reject the no-difference hypothesis under the model's assumptions. It does not mean an observed winner has an 80 percent chance of being real.
Gotcha: '10 percent lift' is ambiguous. From a 5 percent baseline, a 10 percent relative lift ends at 5.5 percent, while a 10 percentage-point lift ends at 15 percent, and their required samples are nowhere close. Account-level randomization, repeated observations, uneven allocation, attrition, and multiple variants also change the calculation. Evan Miller's bookmarked calculator is fine arithmetic and still wrong for you if it assumes independent users and your teams share one account.
Ask AI for it
Calculate the required sample size for this two-variant conversion test with baseline [rate], minimum detectable effect [state absolute or relative], alpha 0.05, power 0.80, and equal allocation. Use Python statsmodels.stats.proportion.proportion_effectsize and statsmodels.stats.power.NormalIndPower.solve_power, show the converted treatment rate and sample size per variant, then divide by measured daily eligible users to estimate whole weeks of runtime. Add 10 percent for expected attrition only if I provide evidence for that attrition rate. Also show a sensitivity table for 5, 10, and 20 percent relative lift, and state every assumption.