How likely is even one false positive across 20 tests?

Question

You run m = 20 independent hypothesis tests, each at level \alpha = 0.05, and every one is actually a true null (no real effect). What is the probability of getting at least one false positive, and how do the Bonferroni and Sidak corrections drive it back down to 5\%?

Accepted Answer

The compounding. Each test individually avoids a false positive with probability 1 - \alpha = 0.95. Across 20 independent true nulls, the chance of no false positive is 0.95^{20}, so the family-wise error rate (FWER), the probability of at least one, is 	ext{FWER} = 1 - (1 - \alpha)^m = 1 - 0.95^{20} \approx 1 - 0.358 = 0.642. So testing 20 dead signals at 5\% each, you are about 64\% likely to "discover" at least one. This is the multiple-comparisons trap: run enough tests and spurious significance is nearly guaranteed. Bonferroni. To force FWER \le \alpha, test each hypothesis at the shrun…

How likely is even one false positive across 20 tests?

Your answer

More Statistics questions