Holdout group
A slice of eligible users deliberately left on business as usual, so their results reveal the baseline without the campaign or rollout.
See it
What it is
A holdout group is a deliberately untouched slice of an eligible audience. Everyone else may receive a campaign, recommendation system, or bundle of product changes; the holdout stays on business as usual. Its outcomes estimate the baseline you cannot observe for the treated people at the same time.
Every randomized experiment has a control, but 'holdout' usually means a reserved group kept out for longer or across several interventions. Reach for one when you need the combined incremental effect of a channel or program, not the winner of one button-color test. Assign it before eligibility or delivery outcomes can influence membership, and keep the assignment stable for the measurement window.
Gotcha: exclusion on paper is not exclusion in reality. A held-out user may see the campaign on another device, share an account with a treated user, or receive the same offer through another channel. That contamination usually shrinks the measured effect. Small holdouts also save reach at the cost of power, so calculate their size before launching rather than picking a painless-looking one percent.
Ask AI for it
Create a durable 10 percent holdout for this lifecycle program using a PostHog feature flag with variants holdout and eligible. Assign by account_id with a stable hash, evaluate the flag before any campaign eligibility rule, and write the assigned variant plus assignment_time to the warehouse. Exclude holdout accounts from every Braze Canvas in the program, but include both groups in outcome queries. Analyze conversion and revenue by original assignment using intention-to-treat, report contamination and missing-assignment rates, and add a test proving the same account stays held out across devices.