How many dead signals will look "significant" by luck?

Question

A quant screens 60 candidate signals, each judged by its own test at the 5% significance level. Suppose that in truth every one of them is worthless (no real edge at all).

On average, how many of these 60 dead signals will still come out "significant"?

Accepted Answer

What a 5% test really promises. Running a test "at the 5% level" means: even when a signal is pure noise, there is a 5% chance it slips through and looks significant anyway. That is the built-in false-alarm rate of the test. You cannot avoid it, it is the price of using a 5% cutoff. Turn one signal into many. If each worthless signal has a 5% chance of falsely lighting up, then across 60 of them the expected number of false alarms is just the count times the per-test chance: 	ext{expected false positives} = m 	imes \alpha = 60 	imes 0.05 = 3. So on average three of the sixty noise signals …

How many dead signals will look "significant" by luck?

Your answer

More Statistics questions