Twelve backtests, all noise: odds of at least one "win"?

Question

A quant runs 12 independent strategy backtests, each judged at the 5% significance level. Suppose all 12 strategies are truly worthless.

What is the probability that at least one of them comes out "significant" anyway?

Accepted Answer

Attack the opposite event. "At least one false alarm" is awkward to count directly, but its opposite is easy: "no false alarms at all," meaning every one of the 12 tests correctly stays quiet. Each test stays quiet with 95% chance. A single null test at the 5% level is quiet 95% of the time. Because the 12 backtests are independent, the chance that all 12 stay quiet is 0.95 multiplied by itself 12 times: P(	ext{all quiet}) = 0.95^{12} \approx 0.540. Flip back. The chance of at least one false alarm is therefore P(	ext{at least one}) = 1 - 0.95^{12} \approx 1 - 0.540 = 0.460, about 46%. Read…

Twelve backtests, all noise: odds of at least one "win"?

Your answer

More Statistics questions