Killing a subject-line test the instant it shows a winner

Question

A marketing team runs A/B tests on email subject lines, measuring open rate. Their tool updates significance in real time as recipients open the email through the day. Their standard practice is to stop the test and roll out the leading subject line as soon as it reaches $p < 0.05$ . They notice that many of these "proven winners" fail to beat the control when reused later.

Explain why their practice produces winners that do not repeat, and what they should change.

Accepted Answer

Real-time stopping is repeated peeking. Opens arrive continuously, and the tool recomputes significance constantly. Stopping at the first moment p < 0.05 means the team is effectively taking the minimum p-value over hundreds of checks. Even when two subject lines are truly equal, random variation in who opens when will drag the running result across the 0.05 line at least once with high probability. So a large fraction of their "significant" wins are pure noise crossings, not real differences. That is exactly why they do not reproduce. The winner's curse compounds it. Stopping the instant a l…

Killing a subject-line test the instant it shows a winner

Your answer

Discussion

Learn the concepts

Related questions