Fifty tests at the strict 1% level

Question

Wary of multiple testing, an analyst uses a strict 1% level and runs 50 independent tests, all of which are truly null.

Find the probability that at least one test is significant, and the per-test threshold that would hold the family-wise false-positive chance to 5%.

Accepted Answer

Probability of at least one false positive. Each truly-null test fails to fire with probability 0.99, and with independence the chance none of the 50 fire is 0.99^{50}. So \mathbb{P}(	ext{at least one}) = 1 - 0.99^{50} = 1 - 0.6050 \approx 0.395 \approx 0.39. So even a tight 1% bar, spread over 50 tests, leaves about a 39% chance of at least one false positive. Lowering alpha helps, but volume still erodes it. (Expected false positives: 50 	imes 0.01 = 0.5.) The Bonferroni fix for a 5% family-wise rate. To keep the chance of any false positive at 5% across 50 tests: \alpha{	ext{each}} = \f…

Fifty tests at the strict 1% level

Your answer

Discussion

Learn the concepts

Related questions