Twenty peeks at a checkout test and a suspiciously fast win

Question

To build trust in their A/B tooling, an e-commerce team runs an A/A test: both arms get the exact same checkout flow, so any measured difference is pure noise. They watch the conversion dashboard once a day for 20 days and record whenever the p-value dips below 0.05. To their surprise, it crosses 0.05 on two separate days.

Explain why an A/A test with no real difference still produces "significant" days, and roughly how likely at least one false crossing is over 20 daily looks.

Accepted Answer

No real difference does not mean no random difference. Even when both arms are identical, day-to-day sampling variation makes the measured conversion gap wobble around zero. On any single planned look, that wobble crosses the 0.05 line about 5% of the time. That is the definition of a 5% false-positive rate. The A/A test is doing exactly what statistics predicts: occasionally, noise alone looks "significant." Twenty looks stack the odds. The trouble is looking 20 times and treating any crossing as a finding. As a rough upper bound, if the looks behaved independently the chance of at least one…

Twenty peeks at a checkout test and a suspiciously fast win

Your answer

Discussion

Learn the concepts

Related questions