How many backtests beat t = 2 by luck?

Question

A researcher backtests 250 strategy variants and reports the winner: a Sharpe with a t-statistic of $2.1$ , "significant at the $5\%$ level."

If none of the 250 had any real edge, how many would you expect to clear a $t$ -stat of $2$ by chance? What does that say about the reported winner?

Accepted Answer

The expected count of lucky winners. Under the null, a t-statistic behaves roughly like a standard normal, and \mathbb{P}(t 2) \approx 4.6\% (about 5\% at the conventional two-sided level). With 250 independent dead strategies, the expected number that clear the bar is 250 	imes 0.05 \approx 12.5. So even if every strategy is worthless, you would expect roughly a dozen to post t-stats above 2. The probability that at least one does is essentially certain: 1 - 0.95^{250} \approx 1 - 3 	imes 10^{-6} \approx 1. What the winner is worth. A single t = 2.1 selected as the best of 250 is not evide…

How many backtests beat t = 2 by luck?

Your answer

More Statistics questions