Twenty signals, one spurious discovery

Question

A researcher tests 20 independent candidate signals, each with a 5%-level test. Assume in truth none of them has any edge.

What is the probability at least one comes out "significant"? How should the testing procedure be corrected, and what are the trade-offs among the standard corrections?

Accepted Answer

The family-wise blow-up. Each null test is a 5% coin. With 20 independent tests, \mathbb{P}(	ext{at least one significant}) = 1 - 0.95^{20} \approx 1 - 0.358 = 0.64. A 64% chance of at least one "discovery" from pure noise. The expected number of false positives is 20 	imes 0.05 = 1, so the typical outcome of scanning 20 dead signals is exactly one exciting result. Bonferroni correction. To keep the family-wise error rate (FWER, the chance of any false positive) at 5%, test each signal at level \alpha/m = 0.05/20 = 0.0025. By the union bound, \mathbb{P}\Big(\bigcup{i=1}^{m} 	ext{FP}i\Big) …

Twenty signals, one spurious discovery

Your answer

More Statistics questions