Why stopping an A/B test at first significance backfires

Question

A subscription news app is testing a redesigned paywall against the old one, with subscription rate as the metric. The plan was to run for 21 days. The analyst opens the dashboard every morning and intends to call a winner the first day the p-value drops below 0.05. On day 4, treatment is ahead with $p = 0.04$ , and the team wants to ship.

What is wrong with declaring victory now, and what should the team have done instead?

Accepted Answer

The 5% promise applies to one look, not many. When you fix your sample size in advance and check the result exactly once, a design at \alpha = 5\% says: if the two paywalls are truly identical, the data will falsely cross the 0.05 line only 5% of the time. That guarantee is attached to the single planned look. Peeking changes the procedure. Checking every day and stopping the first time p < 0.05 is a completely different game. Think of the running result as a wandering value that bounces around due to random noise. Each new day is another chance for that wandering value to stray across the li…

Why stopping an A/B test at first significance backfires

Your answer

Discussion

Learn the concepts

Related questions