Quant Memo
Statistics/●●●●●

Killing a subject-line test the instant it shows a winner

A marketing team runs A/B tests on email subject lines, measuring open rate. Their tool updates significance in real time as recipients open the email through the day. Their standard practice is to stop the test and roll out the leading subject line as soon as it reaches p<0.05p < 0.05. They notice that many of these "proven winners" fail to beat the control when reused later.

Explain why their practice produces winners that do not repeat, and what they should change.

Your answer

This one is open-ended. Work it through, then check your reasoning against the full solution.

More Statistics questions