Twenty strategies, one bar, and the odds of a fluke

Question

A desk screens 20 independent candidate strategies, all of which are in truth worthless, testing each at the 5% significance level.

Find (a) the expected number that look significant, (b) the probability that at least one looks significant, and (c) the per-test threshold that would hold the chance of any false positive to 5%.

Accepted Answer

(a) Expected number of false positives. Each worthless strategy is flagged with probability 0.05, so across 20 the expected count is 20 	imes 0.05 = 1. You should expect one false discovery even though nothing works. (b) Probability of at least one false positive. With independent tests, the chance that a given test does not fire is 0.95, so the chance that none of the 20 fire is 0.95^{20}. Therefore \mathbb{P}(	ext{at least one}) = 1 - 0.95^{20} = 1 - 0.3585 \approx 0.6415 \approx 0.64. So there is about a 64% chance of finding at least one "significant" strategy purely by luck. The family…

Twenty strategies, one bar, and the odds of a fluke

Your answer

More Statistics questions