Survivorship bias
The mistake of learning only from users who stayed, while the people who quit or failed are missing from the evidence.
See it
What it is
Survivorship bias happens when the people or things that remain visible stand in for the full group, while failures and departures vanish from the evidence. Ask today's power users which onboarding steps they used and you learn about people who survived onboarding, not everyone who started it. The missing users may be missing precisely because those steps failed them.
The classic anchor is Abraham Wald's analysis of damage on returning World War II aircraft. Armor was needed where the surviving planes had no bullet holes, because planes hit there did not return to be counted. Reach for the same move in product data: begin with the original signup or eligible cohort, preserve zero-activity rows, and compare retained, churned, refunded, and deleted outcomes instead of querying only current accounts.
Gotcha: adding more detail about survivors does not recover the missing population. Churn surveys have the same bias if only unusually motivated former users answer. Track the denominator and loss at each stage; label right-censored observations when the outcome has not had time to occur; and say when privacy deletion or old instrumentation makes the missing history unknowable.
Ask AI for it
Audit this retention analysis for survivorship bias. Start from the canonical signups table and LEFT JOIN activity, subscription, refund, and deletion-status records so users with no later events remain as zeroes instead of disappearing. Print the original cohort size and the count lost at every filter. For time-to-churn, build a Kaplan-Meier estimate with Python's lifelines.KaplanMeierFitter, passing explicit event and right-censoring flags rather than averaging tenure among active accounts. Compare retained and churned users, list data that cannot be recovered, and remove every WHERE clause that silently requires a post-signup event.