Why significant backtests overstate the edge

Question

A researcher runs many low-powered backtests and reports only the ones that pass $p < 0.05$ (two-sided).

Explain why the reported edges systematically overstate the true edge even when some effects are real, and quantify the inflation with a concrete example where the true effect equals one standard error.

Accepted Answer

The mechanism. A backtest reports a "significant" edge only if its estimate is at least about 1.96 standard errors from zero. When the test has low power, a real effect crosses that bar only on the lucky draws where noise happened to push the estimate up. So conditioning on significance selects the upward-biased tail of the estimates. The published edge is therefore larger than the truth. This is the winner's curse (also called a Type M, or magnitude, error). A concrete example. Suppose the true edge equals exactly one standard error, so the estimate is distributed as N(1, 1) in standard-erro…

Why significant backtests overstate the edge

Your answer

More Statistics questions