What the 5% significance level actually promises

Question

A research desk uses the usual 5% significance level ( $\alpha = 0.05$ ) for every test. Suppose they run this exact test on 200 candidate strategies that, unknown to them, all have zero real edge (the null is true for every one).

About how many of these 200 dead strategies will still come back flagged as "statistically significant," and what does that number tell you about the meaning of $\alpha$ ?

Accepted Answer

The calculation. The significance level \alpha is the probability of wrongly rejecting the null when the null is actually true. That is exactly the situation here: every strategy is worthless, so every "significant" flag is a false alarm. Each test fires a false alarm with probability 0.05, so out of 200 tests you expect 200 	imes 0.05 = 10 strategies to look significant purely by luck. What alpha means. This is the plain meaning of a 5% significance level: if you keep testing things that have no real effect, about 1 in 20 will cross the bar anyway. Alpha is a long-run false-alarm rate you a…

What the 5% significance level actually promises

Your answer

Discussion

Learn the concepts

Related questions