Thirty questions, all null: almost sure to find "something"

Question

A market-research survey tests 30 independent hypotheses (30 different "does X drive Y?" questions), each at the 5% significance level. Suppose every one of them is truly null.

What is the probability that at least one comes out "significant"?

Accepted Answer

Use the complement. Rather than count all the ways "at least one" can happen, count the single way it fails: every one of the 30 tests correctly stays quiet. All thirty stay quiet. A null test is quiet with probability 0.95. With 30 independent tests, all staying quiet has probability 0.95^{30} \approx 0.215. So at least one fires. The chance of at least one false positive is 1 - 0.95^{30} \approx 1 - 0.215 = 0.785, roughly 79%. What it means. With 30 questions, it is nearly guaranteed you will find "something significant" even if nothing is real. A survey that proudly reports its one or two …

Thirty questions, all null: almost sure to find "something"

Your answer

More Statistics questions