Parameter Sensitivity Analysis

Almost every strategy has parameters, a moving-average lookback of 20 days, an entry threshold of 2 standard deviations, a rebalance frequency of weekly. It's tempting to search over many parameter values, pick whichever combination produced the best backtest Sharpe ratio, and call that the strategy. Parameter sensitivity analysis asks a different question: not "what's the single best setting?" but "how much does performance change as I move a little away from that setting?" A strategy whose performance collapses the moment a parameter shifts from 20 days to 22 days was never robust, it was fit to the specific noise in the historical sample.

Peaks versus plateaus

Picture backtest Sharpe ratio plotted against a parameter's value. A robust strategy produces a broad plateau: performance stays reasonably good across a wide range of nearby settings, meaning the edge doesn't depend on hitting one exact number. An overfit strategy produces a sharp, narrow peak: performance is great at one specific value and falls off a cliff just next to it. That sharp peak is the signature of a parameter that was tuned to exploit specific quirks in the historical sample rather than a genuine, persistent market relationship, real market relationships tend to be smooth and forgiving of small parameter changes, since they aren't calibrated to the idiosyncrasies of one dataset.

Worked example: a moving-average crossover strategy

A trend-following strategy uses a fast and slow moving-average crossover, and a grid search over fast lengths (5–40 days) and slow lengths (50–200 days) finds the best backtest Sharpe, 1.4, at fast=17, slow=83. Testing the eight neighboring combinations (fast 15–19, slow 78–88) gives Sharpes of 0.3, 0.4, 1.4, 0.5, 0.2, 0.6, 0.3, 0.4, the 1.4 is an isolated spike surrounded by mediocre-to-poor results. Compare that to a second candidate whose best setting (fast=20, slow=100) scores 1.1, with neighboring combinations scoring 0.9 to 1.2 across the board. The second strategy has a lower peak but a much broader plateau, and is the more trustworthy candidate going forward, the first strategy's 1.4 is very likely a product of the grid search finding one lucky combination among dozens tested, not a real edge at fast=17, slow=83 specifically.

What this means in practice

Sensitivity analysis is one of the simplest, cheapest checks a researcher can run before trusting a backtest result: rerun the strategy across a grid of parameter values near the reported optimum and look at the whole surface, not just the peak. Many teams intentionally set a strategy's live parameters slightly away from the exact historical optimum, closer to the middle of a broad plateau, precisely because the plateau's center is more likely to be robust than its edge or an isolated spike. If a strategy has many parameters, the number of possible combinations grows fast, and a global search over all of them increases the chance of finding a spurious peak purely by luck, a risk closely related to the multiple-testing problem in backtest overfitting.