Statistical Power Analysis for Backtests

Imagine testing a strategy with a real Sharpe ratio of 0.3 on only six months of daily data. Even if the edge is genuinely there, six months of noisy daily returns is often not enough to distinguish that Sharpe from zero — the test will come back "not significant" most of the time, and you'll throw away a real strategy thinking it doesn't work. That's the blind spot most backtest reviews skip: everyone checks whether a result is significant, almost nobody checks whether the test had any real chance of detecting the effect in the first place. That chance is called statistical power.

The idea: could this test even have found it?

Power is the probability that a hypothesis test correctly rejects the null when the alternative is actually true — in backtest terms, the probability you'd correctly flag a strategy as having a real edge, given the sample size and how large that edge truly is. Power depends on three things pulling against each other: a bigger true effect is easier to detect, more data makes any given effect easier to detect, and a stricter significance threshold makes everything harder to detect. Low power means a "no" from your backtest is close to meaningless — it could be a genuinely dead strategy, or a good strategy tested on too little data.

Worked example: how much data do you need?

Suppose you want to detect a true annualized Sharpe ratio of $SR = 0.5$, testing at the conventional 5% significance level, and you want 80% power (the standard target). A widely used approximation for the number of years of daily data needed is:

$$n \approx \left(\frac{z_{\alpha} + z_{\beta}}{SR}\right)^2$$

where $z_\alpha \approx 1.96$ is the critical value for a 5% two-sided test and $z_\beta \approx 0.84$ is the value corresponding to 80% power. Plugging in:

$$n \approx \left(\frac{1.96 + 0.84}{0.5}\right)^2 = \left(\frac{2.8}{0.5}\right)^2 = 31.4 \text{ years.}$$

In plain English: to reliably detect a true Sharpe of 0.5 with standard significance and power targets, you'd need over three decades of data — far more than almost any backtest has. A strategy tested on 3 years with a true Sharpe of 0.5 will fail to reach significance most of the time, not because the edge isn't real, but because the test was underpowered from the start.

What this means in practice

Power analysis should run before a backtest, not after a disappointing one: decide what effect size you actually care about detecting, then check whether the available history gives the test a fighting chance. When power is low, a null result tells you almost nothing — treat it as inconclusive rather than as proof the strategy fails. This matters especially for niche or short-history markets (a new token, a recently listed ETF) where a genuinely profitable strategy can look statistically indistinguishable from noise simply because there isn't enough data yet, not because the edge isn't there.