Why you can't stop an A/B test when it "hits significance"

Question

You planned a 14-day A/B test of a new execution algo. On day 3, the dashboard shows the treatment ahead with $p = 0.03$ , and your PM wants to declare victory and ship.

What is wrong with stopping now, and what are the statistically sound alternatives?

Accepted Answer

The problem: optional stopping. A fixed-sample test at \alpha = 5\% guarantees that if you look once, at the planned end, a null effect crosses the threshold 5% of the time. Monitoring continuously and stopping the first time p < 0.05 is a different procedure: you are taking the minimum p-value over many looks. Under the null, the test statistic is a random walk that wanders; the more times you check, the more chances it has to wander across the boundary at least once. The realized false-positive rate is therefore strictly greater than 5%, with daily peeking over a couple of weeks it lands in…

Why you can't stop an A/B test when it "hits significance"

Your answer

More Statistics questions