A/B test (split test)
Show version A to half your visitors and version B to the other half at random, then keep whichever wins on one agreed metric.
See it
What it is
An A/B test is a controlled experiment on live traffic. Random assignment is the whole trick: because the split is random, the only systematic difference between the two groups is the change you made, so a gap in results can fairly be blamed on it. Change two things at once and you learn that something worked, not what.
Decide before launch: the one metric that picks the winner, the smallest lift worth caring about, and the sample size that gets you there. Then leave it alone until it hits that number. Usual tooling: Optimizely, VWO, PostHog experiments, GrowthBook, or a plain feature flag plus event logging if you already have both.
Gotcha: peeking. Checking every morning and stopping the moment your variant is ahead manufactures winners out of noise, which is where most 'we lifted conversion 40%' case studies come from. Run whole weeks too, because Tuesday traffic and Saturday traffic are different people.
Ask AI for it
Design an A/B test for [change] on [page]. Give me: the hypothesis written as 'because X, changing Y will improve Z', the single primary metric plus two guardrail metrics, and the exact copy or markup for control and variant. Then, using baseline conversion [rate], minimum worthwhile lift [absolute or relative lift], weekly eligible traffic [n], alpha 0.05, and 80% power, calculate the sample size per arm and the estimated runtime, showing the formula you used. Do not invent missing inputs: if one of those numbers is absent, name it and stop rather than guessing. Finish with a stop rule that forbids ending early. Vary one element only, and list everything that must stay identical across both variants.