Two hundred A/B variants, all useless, at the 1% level

Question

A product team runs 200 separate A/B tests over a quarter, each evaluated at a strict 1% significance level. Suppose none of the 200 changes actually improves anything.

On average, how many of these 200 useless changes will nonetheless test as a "significant" winner?

Accepted Answer

The per-test false-alarm rate. A 1% significance level means that even a change with zero real effect has a 1% chance of looking significant by luck. That is the false-positive rate baked into the test. Scale it up. Across 200 such tests, the expected number of false alarms is the count multiplied by the per-test chance: 	ext{expected false positives} = m 	imes \alpha = 200 	imes 0.01 = 2. So even with a strict 1% bar, you should expect about two phantom winners from a batch of 200 truly useless changes. The lesson. A tight significance level slows the false alarms but does not stop them. …

Two hundred A/B variants, all useless, at the 1% level

Your answer

Discussion

Learn the concepts

Related questions