Sensitivity to Universe Definition

A factor strategy backtested on stocks above $500 million market cap shows a Sharpe ratio of 1.4. Move the cutoff to $400 million and it's 1.3. Move it to $300 million and it's 1.35. Move it to $600 million and it's 0.6. That last jump should worry a researcher far more than a strategy that stayed roughly steady across every reasonable cutoff — because the $500 million line in the original backtest wasn't chosen for any economic reason, it was one of infinitely many defensible thresholds, and a result that depends this heavily on which one was picked is a result that depends on an arbitrary modeling decision rather than a real, broad-based pattern.

Why the universe is a hidden parameter

Every backtest implicitly optimizes over its universe definition even when nobody explicitly tunes it, because a researcher trying several plausible cutoffs during development and keeping the one that "worked best" is doing exactly the same kind of overfitting as tuning any other parameter — it's just less visible, because "$500 million felt like a reasonable cutoff" doesn't look like an optimization the way "I tried 40 lookback windows" does. Universe definitions have many hidden degrees of freedom: market-cap or size cutoffs, minimum price filters (to exclude penny stocks), minimum average daily volume (to exclude illiquid names), exchange or listing-country restrictions, and sector or industry exclusions. Each one is a lever, and a strategy that was implicitly shaped by trying several settings of each lever is more overfit than its backtest report will show.

Worked example: sweeping the market-cap cutoff

A strategy's Sharpe ratio is recomputed at market-cap cutoffs of $200m, $300m, $400m, $500m, $600m, $800m: $1.1, 1.25, 1.3, 1.4, 0.6, 0.65$. The strategy is clearly robust across the $200m-$400m range, roughly stable through $500m, and then falls off a cliff at $600m and stays low. If the original research settled on the $500m cutoff specifically, that choice sits right at the edge of the stable region — a small, entirely plausible change to the cutoff (say, using a slightly different data vendor's market-cap figures, which routinely differ by 10-20%) could push some names across the $500-$600m boundary and materially change the realized live performance. A researcher who only reports the $500m result, without showing this sweep, is hiding exactly the information a reviewer needs to judge how fragile the result really is.

What this means in practice

Report a sensitivity sweep over the universe definition alongside the headline backtest result as standard practice, the same way a parameter sensitivity sweep is standard for any other tunable threshold. When performance is stable across a wide range of reasonable universe definitions, that's genuine evidence of robustness. When it's not, the honest conclusion is that the strategy's edge, as measured, is at least partly an artifact of where the universe boundary happened to be drawn — and the live version should either use a cutoff comfortably inside the stable region, or the whole result should be treated with real skepticism.