Incrementality test

An incrementality test randomly withholds a channel or campaign from an eligible group, then compares its outcomes with the group allowed to receive it. If the treated group converts at 6 percent and the holdout at 5 percent, the measured incremental lift is 1 percentage point. The other conversions were likely to happen without the campaign, even if an attribution report credited the campaign for them.

Reach for it when deciding whether to keep spending on paid search, retargeting, discounts, emails, or any channel that mostly reaches people already close to buying. The famous case is eBay: when its economists switched paid search off in a controlled 2012 experiment, most of the attributed revenue kept arriving anyway. Randomize people when exposure can be controlled per person; use matched geographic regions when a channel such as television cannot be cleanly switched off at that level. Measure assignment, not only who happened to see an ad.

Gotcha: the effect is often much smaller than attributed conversions, so these tests need more traffic and time than a dashboard suggests. Cross-device exposure, neighboring geographies, and simultaneous campaigns contaminate the holdout. A null result with a wide confidence interval does not prove zero lift; it says the test could not separate the plausible lift from noise.