Power, the miss rate, and a real edge

Question

A backtest has 75\% power to detect a real edge of the size you care about. If the edge is genuinely there, what is the probability this single test fails to flag it (its type II error rate)?

Accepted Answer

Power and the miss rate are complements. Power is the probability of correctly rejecting the null when the effect is real; the type II error rate \beta is the probability of failing to reject in that same situation. Together they cover all outcomes when the effect is real, so \beta = 1 - 	ext{power} = 1 - 0.75 = 0.25. So even with a genuine edge, a test powered at 75% will miss it about one time in four. That is why "the backtest was not significant" is not the same as "there is no edge": a modestly powered test fails to flag real effects fairly often. Takeaway. The miss rate is just \beta =…

Power, the miss rate, and a real edge

Your answer

More Statistics questions