Train-Test Decay Curves for ML Signals

Two models can have identical in-sample fit and identical one-month-ahead validation scores, and yet behave completely differently a year after their last refit — one keeps predicting reasonably well, the other's accuracy collapses within weeks. You can't tell the difference from a single validation number computed right after training. A train-test decay curve makes the difference visible directly: it plots model performance as a function of how much time has elapsed since training, so you can see the shape of the decay, not just one point on it.

Building the curve

Train a model once on data up to some cutoff date, then evaluate it separately on test windows at increasing distances from that cutoff — say, months 1, 2, 3, up to 12 after training ended — instead of pooling all post-training data into one aggregate score. Plotting the resulting performance metric (information coefficient, accuracy, or Sharpe ratio of a resulting strategy) against elapsed time since training gives the decay curve. In plain English: rather than asking "how good is this model on unseen data," ask "how good is this model at 1 month unseen, at 6 months unseen, at 12 months unseen," because those can be very different answers even though all three points are technically "out-of-sample."

Repeating this for several different training cutoffs and averaging the resulting curves smooths out the effect of any one lucky or unlucky historical period, giving a more reliable picture of the typical decay shape rather than one particular episode's.

Worked example: two signals with the same validation score, different decay

Signal A and Signal B both show an information coefficient of 0.05 in the first month after training. Tracing the full decay curve: Signal A's IC falls gradually — 0.05, 0.045, 0.04, 0.035 — over the first four months, staying usefully positive for roughly a year. Signal B's IC falls off a cliff — 0.05, 0.02, 0.005, near zero — by month three, essentially useless from month four onward. If a researcher only checked the one-month score, both would look identical and equally worth deploying. The decay curves reveal that Signal A tolerates a slower, cheaper refit schedule (say, quarterly), while Signal B needs a much more frequent refit — monthly or even more often — to stay useful, or it needs to be abandoned if that refit cadence isn't operationally feasible.

What this means in practice

Decay curves directly inform two practical decisions: how often to refit (a curve that decays slowly tolerates infrequent, cheaper refits; a steep curve demands near-continuous retraining or should raise doubts about whether the signal is worth deploying at all), and how much to trust a signal's staying power when sizing a position or committing research effort to it. They also serve as an early-warning validation tool: if a live model's realized decay curve looks much steeper than the curve estimated during research, that's a sign either the original estimate was optimistic or the market relationship genuinely shifted.