Parameter Stability Over Time

Optimize a moving-average crossover strategy on 2019-2020 data and the best lookback is 12 days. Re-run the identical optimization on 2021-2022 data and the best lookback is 47 days. Re-run it again on 2023-2024 and it's 8 days. A parameter that means something real about market structure should be reasonably stable across time periods; a parameter that bounces around this much is a sign the "optimal" value each time was mostly fitting noise specific to that window, not capturing a persistent relationship the strategy can actually rely on going forward.

Testing stability directly

The test is simple to run: split history into several non-overlapping windows (by year, by quarter, or using a rolling re-optimization scheme), re-run the parameter optimization independently on each window, and look at how much the resulting best-fit parameter moves around. A stable parameter clusters tightly across windows — the underlying market relationship it's capturing didn't change much, so neither did the number that best captures it. An unstable parameter swings widely and unpredictably from window to window, which is the signature of overfitting: each window's "optimal" value is really just whatever value happened to fit that window's particular noise best, and noise doesn't repeat from one period to the next. A genuinely stable parameter behaves like a mean-reverting process — it wanders a little from window to window because real data is noisy, but it keeps returning to roughly the same neighborhood rather than drifting or jumping to an unrelated level each time.

Worked example: two parameters, two verdicts

A momentum strategy's lookback window, re-optimized on five non-overlapping two-year windows, comes out as: 58, 61, 55, 64, 59 days. These cluster tightly around a mean of about 59 with a standard deviation of roughly 3 days — a stable parameter, and a reasonable case that 59-60 days captures something real about how momentum decays in this market. A second strategy's stop-loss threshold, re-optimized the same way, comes out as: 1.2%, 4.8%, 0.6%, 3.9%, 2.1%. These range across the full plausible space with no discernible cluster — this parameter is not stable, and any single "optimal" value chosen from a full-history fit should be treated with much less confidence, or replaced with a value chosen on economic reasoning rather than optimization.

What this means in practice

An unstable parameter doesn't necessarily mean the strategy is worthless — it means the specific optimized value can't be trusted, and the strategy should either use a value chosen on independent economic reasoning, be re-optimized on a rolling basis with the understanding that "optimal" will keep drifting, or be simplified to remove the unstable parameter entirely. Parameter stability checks are cheap relative to the damage an overfit parameter can do live, and they're one of the fastest ways to separate a genuinely robust strategy idea from one that only looks good because it was tuned to fit a specific historical window.