60 neutral button colors at the 5% level

Question

A growth team runs 60 A/B tests, each comparing a new button color against the current one, at the 5% significance level. Suppose button color genuinely has no effect on conversion in any of the 60 tests.

About how many of these 60 tests will crown the new color a "significant winner," and what does that say about $\alpha$ ?

Accepted Answer

The calculation. If color has no real effect, the null is true in all 60 tests. Each test wrongly declares a winner with probability equal to the significance level, 0.05. So you expect 60 	imes 0.05 = 3 false "winners." What alpha means. Even though nothing works, roughly 3 tests will show a "significant" lift purely from random variation in who clicked. Alpha is the long-run rate at which truly-null experiments produce a false positive. It is a property of the testing procedure, not proof about any single test. The trap. If the team ships all 3 "winning" colors, they will see the lift vani…

60 neutral button colors at the 5% level

Your answer

Discussion

Learn the concepts

Related questions