Five null tests at a loose 10% level

Question

An analyst runs just 5 independent tests, but at a loose 10% significance level. Suppose all 5 hypotheses are truly null. What is the probability that at least one comes out "significant"?

Accepted Answer

Flip to the easy event. "At least one false alarm" is the opposite of "all five stay quiet," so compute the quiet case first. All five quiet. At the 10% level, a null test stays quiet with probability 1 - 0.10 = 0.90. Across 5 independent tests, 0.90^{5} = 0.59049 \approx 0.590. At least one fires. Therefore 1 - 0.90^{5} \approx 1 - 0.590 = 0.410, about 41%. The point about the bar. Only five tests still gives a 41% false-alarm risk, because the 10% level is lax. Both levers matter: how many tests you run and how loose each test is. A generous significance level multiplies quickly once you ru…

Five null tests at a loose 10% level

Your answer

More Statistics questions