Eight null tests at 5%: what's the false-alarm risk?

Question

A team evaluates 8 independent candidate features for a model, each with a 5%-level test. Suppose none of the 8 features actually helps.

What is the probability that at least one tests as "significant"?

Accepted Answer

Turn it around. Instead of counting "at least one false alarm," count its opposite: all 8 tests correctly stay quiet. All eight quiet. A null test at the 5% level stays quiet with probability 0.95. With 8 independent tests, 0.95^{8} \approx 0.663. At least one fires. So the chance of at least one false positive is 1 - 0.95^{8} \approx 1 - 0.663 = 0.337, about 34%. Interpreting it. Just eight useless features carry a one-in-three chance of producing a "significant" one. It grows fast: at 12 tests the risk is about 46%, at 20 tests about 64%. Every extra test you run quietly spends more of your…

Eight null tests at 5%: what's the false-alarm risk?

Your answer

More Statistics questions