Twenty backtests, one hit at p

Question

A researcher runs 20 independent backtests, each testing a different worthless-in-truth idea at significance level $\alpha = 5\%$ . One of them comes back with $p < 0.05$ and gets pitched as a discovery.

If all 20 ideas were truly worthless, how many false positives would you expect, and what is the chance of getting at least one? What correction fixes this?

Accepted Answer

Expected false positives. Under the null (every idea worthless), each test has a 5\% chance of falsely clearing the bar. With 20 independent tests, the expected number of false positives is 20 	imes 0.05 = 1. So even if nothing works, you expect one "significant" result just from noise. Finding one hit in 20 tries is exactly what pure chance predicts, not evidence of anything. Probability of at least one false positive. Each test independently avoids a false positive with probability 0.95, so all 20 stay quiet with probability 0.95^{20}: P(	ext{at least one significant}) = 1 - 0.95^{20} = 1…

Twenty backtests, one hit at p < 0.05

Your answer

More Statistics questions