Romano-Wolf Stepwise Multiple Testing

If a researcher backtests a hundred strategy variants and picks the one with the best Sharpe ratio, some of that outperformance is just luck from testing so many candidates — the multiple testing problem. A simple fix like Bonferroni correction divides the significance threshold by the number of tests, but it's overly harsh because it doesn't use the fact that the strategies' returns are correlated with each other; a genuinely strong strategy can get rejected just because it was tested alongside ninety-nine noisy siblings.

Romano-Wolf testing fixes this by resampling the actual return data (via bootstrap) to estimate how correlated the test statistics really are, and then testing strategies stepwise: the single best-performing strategy is tested first against a threshold built from the joint distribution of all hundred, and if it survives, it's set aside and the next-best is tested against a threshold built from the remaining ninety-nine, and so on. Each step's threshold accounts for the fact that some strategies have already been confirmed as real, letting the correction loosen up rather than staying uniformly strict.

The net effect is a method with more statistical power than Bonferroni or simple false-discovery-rate corrections, while still properly controlling the chance that any one of the surviving strategies is a false positive — making it a standard tool in academic factor research for testing large batches of candidate signals honestly.