Parameter Sensitivity Analysis

Almost every strategy has parameters — a moving-average lookback of 20 days, an entry threshold of 2 standard deviations, a rebalance frequency of weekly. It's tempting to search over many parameter values, pick whichever combination produced the best backtest Sharpe ratio, and call that the strategy. Parameter sensitivity analysis asks a different question: not "what's the single best setting?" but "how much does performance change as I move a little away from that setting?" A strategy whose performance collapses the moment a parameter shifts from 20 days to 22 days was never robust — it was fit to the specific noise in the historical sample.

Peaks versus plateaus

Picture backtest Sharpe ratio plotted against a parameter's value. A robust strategy produces a broad plateau: performance stays reasonably good across a wide range of nearby settings, meaning the edge doesn't depend on hitting one exact number. An overfit strategy produces a sharp, narrow peak: performance is great at one specific value and falls off a cliff just next to it. That sharp peak is the signature of a parameter that was tuned to exploit specific quirks in the historical sample rather than a genuine, persistent market relationship — real market relationships tend to be smooth and forgiving of small parameter changes, since they aren't calibrated to the idiosyncrasies of one dataset.

Worked example: a moving-average crossover strategy

A trend-following strategy uses a fast and slow moving-average crossover, and a grid search over fast lengths (5–40 days) and slow lengths (50–200 days) finds the best backtest Sharpe, 1.4, at fast=17, slow=83. Testing the eight neighboring combinations (fast 15–19, slow 78–88) gives Sharpes of 0.3, 0.4, 1.4, 0.5, 0.2, 0.6, 0.3, 0.4 — the 1.4 is an isolated spike surrounded by mediocre-to-poor results. Compare that to a second candidate whose best setting (fast=20, slow=100) scores 1.1, with neighboring combinations scoring 0.9 to 1.2 across the board. The second strategy has a lower peak but a much broader plateau, and is the more trustworthy candidate going forward — the first strategy's 1.4 is very likely a product of the grid search finding one lucky combination among dozens tested, not a real edge at fast=17, slow=83 specifically.

What this means in practice

Sensitivity analysis is one of the simplest, cheapest checks a researcher can run before trusting a backtest result: rerun the strategy across a grid of parameter values near the reported optimum and look at the whole surface, not just the peak. Many teams intentionally set a strategy's live parameters slightly away from the exact historical optimum, closer to the middle of a broad plateau, precisely because the plateau's center is more likely to be robust than its edge or an isolated spike. If a strategy has many parameters, the number of possible combinations grows fast, and a global search over all of them increases the chance of finding a spurious peak purely by luck — a risk closely related to the multiple-testing problem in backtest overfitting.