An automated alert that pings the moment p drops below 0.05

Question

An ad-tech team runs A/B tests on click-through rate. To "save time," an engineer wires up a bot that recomputes the p-value after every 1,000 new impressions and automatically halts the experiment and declares treatment the winner the moment $p < 0.05$ . The team loves how fast tests now finish.

Explain why this automated rule dramatically inflates the false-positive rate, and how you would fix it.

Accepted Answer

Continuous checking is the worst case of peeking. The 5% error guarantee assumes a single look at a pre-planned sample size. This bot instead evaluates the result hundreds of times and stops at the first lucky crossing. Under the null (the two ads are truly equal), the running test statistic drifts around like a random walk. Checking it after every batch gives noise an enormous number of chances to cross the 0.05 line at least once. The realized false-positive rate is no longer 5%; with checks this frequent it can climb well above 50%, and in the limit of never-ending checks it tends toward 1…

An automated alert that pings the moment p drops below 0.05

Your answer

Discussion

Learn the concepts

Related questions