Ten backtests and the chance of a false winner

Question

A researcher runs 10 independent backtests, all secretly worthless, each tested at the 5% level.

Find the probability that at least one shows up significant, and the per-test threshold needed to keep the family-wise false-positive chance at 5%.

Accepted Answer

Probability of at least one false positive. Each test fails to fire with probability 0.95, and with independence the chance none of the 10 fire is 0.95^{10}. So \mathbb{P}(	ext{at least one}) = 1 - 0.95^{10} = 1 - 0.5987 \approx 0.4013 \approx 0.40. Even with only 10 dead ideas, there is about a 40% chance of stumbling onto a "significant" one by luck. (The expected number of false positives is 10 	imes 0.05 = 0.5.) The Bonferroni fix. To hold the chance of any false positive to 5% across all 10 tests, tighten the per-test level: \alpha{	ext{each}} = \frac{0.05}{10} = 0.005. A strategy mus…

Ten backtests and the chance of a false winner

Your answer

More Statistics questions