Strict 1% level, 500 dead strategies

Question

A quant team decides to be strict and uses a 1% significance level ( $\alpha = 0.01$ ). They run the test on 500 candidate signals that all secretly have zero edge.

About how many of these 500 dead signals will still be flagged as significant, and what does that tell you about the role of $\alpha$ ?

Accepted Answer

The calculation. With the null true for every signal, each one is wrongly flagged with probability equal to the significance level, 0.01. Across 500 tests you expect 500 	imes 0.01 = 5 false alarms. What alpha means. Lowering \alpha from 5% to 1% cuts the false-alarm rate: now only about 1 in 100 worthless signals slips through, instead of 1 in 20. So \alpha is a dial. Turn it down and you catch fewer flukes, but you also make it harder to detect real effects (you lose power). There is no free lunch. The nuance. A stricter level does not make a single significant result "more true" in isolat…

Strict 1% level, 500 dead strategies

Your answer

More Statistics questions