A hundred tests at a strict 1% still isn't safe

Question

Worried about false alarms, a researcher tightens each test to a strict 1% level, but then runs 100 independent tests. Suppose all 100 hypotheses are truly null.

What is the probability that at least one still comes out "significant"?

Accepted Answer

Complement first. The opposite of "at least one false alarm" is "all 100 tests stay quiet," so start there. All hundred quiet. At the 1% level, a null test is quiet with probability 1 - 0.01 = 0.99. Across 100 independent tests, 0.99^{100} \approx 0.366. At least one fires. So 1 - 0.99^{100} \approx 1 - 0.366 = 0.634, roughly 63%. The uncomfortable lesson. Even a strict 1% bar loses control once you run enough tests: a 63% chance of a false alarm is worse than the 46% you would get from a dozen tests at 5%. Tightening each test helps, but it cannot keep pace with a large number of tests. To t…

A hundred tests at a strict 1% still isn't safe

Your answer

More Statistics questions