Parameter Plateaus Versus Spikes
A backtest's best parameter setting can sit on a broad plateau of nearby settings that all work, or on a narrow spike surrounded by settings that fail. Only the plateau is a real signal.
Prerequisites: Backtest Overfitting
A mean-reversion strategy with a 20-day lookback backtests to a Sharpe of 2.4. That single number tells you nothing about whether the strategy found a real, stable pattern or got lucky at exactly one setting. Run the backtest again at a 19-day and a 21-day lookback — one day either side — and the honest question is whether the Sharpe stays near 2.4 or collapses to 0.3. If it collapses, the "20" wasn't found, it was fit, and it will not survive contact with data the parameter search hasn't seen yet.
Plateau versus spike
A parameter plateau is a region of nearby settings that all produce similar, respectable performance — evidence the strategy is picking up something structural about the market, robust to the exact knob position. A parameter spike is a single setting, or a narrow cluster, surrounded by mediocre or negative performance everywhere else — evidence the search process found one lucky alignment between a parameter and this specific historical sample, with nothing generalisable behind it. Both can report the identical headline Sharpe. Only a scan across neighbouring values tells them apart, and a research process that reports only the single best setting is, by construction, hiding exactly the information needed to tell.
Worked example
Sweep the lookback parameter for the mean-reversion strategy above from 5 to 40 days in steps of 1, backtesting each:
| Lookback (days) | Sharpe |
|---|---|
| 15 | 0.9 |
| 18 | 1.1 |
| 19 | 0.4 |
| 20 | 2.4 |
| 21 | 0.3 |
| 22 | 0.8 |
| 25 | 1.0 |
The neighbourhood of the reported best setting is worse than mediocre on both sides — a spike one day wide. Compare a second strategy, a 20-day moving-average crossover, swept the same way:
| Lookback (days) | Sharpe |
|---|---|
| 15 | 1.2 |
| 18 | 1.5 |
| 19 | 1.6 |
| 20 | 1.7 |
| 21 | 1.6 |
| 22 | 1.5 |
| 25 | 1.3 |
Its reported Sharpe, 1.7, is lower than the mean-reversion strategy's 2.4 — and it is the far more trustworthy number, because every setting within five days of the optimum still clears 1.2. The mean-reversion strategy's higher headline number is the less credible of the two.
Never report a single optimised parameter's performance without the sweep around it. A wide plateau of similar results is evidence of a real pattern; a narrow spike is evidence of overfitting to one specific sample, regardless of how high the peak reads.
The classic confusion is treating "we optimised the parameter and got a great Sharpe" as validation, when parameter optimisation is exactly the process that manufactures spikes. Optimising over more candidate values without a corresponding plateau check makes overfitting more likely, not less — more knobs, more chances for one accidental alignment with the historical sample.
Fast diagnostic: average the Sharpe over a small neighbourhood of the optimum (say, ±10% of the parameter's range) instead of reporting the peak value alone. A strategy whose neighbourhood average is close to its peak is on a plateau; one where the average collapses relative to the peak is on a spike.
A plateau is necessary but not sufficient — it rules out one specific failure mode, single-point overfitting, but says nothing about whether the whole strategy class was cherry-picked from many tried. See Parameter Sensitivity Analysis for the fuller sweep methodology and Combinatorially Symmetric Cross-Validation and PBO for detecting overfitting that a single parameter sweep can miss.
Related concepts
Practice in interviews
Further reading
- Bailey, Borwein, López de Prado & Zhu, Pseudo-Mathematics and Financial Charlatanism