Train-Test Decay Curves for ML Signals

Two models can have identical in-sample fit and identical one-month-ahead validation scores, and yet behave completely differently a year after their last refit, one keeps predicting reasonably well, the other's accuracy collapses within weeks. You can't tell the difference from a single validation number computed right after training. A train-test decay curve makes the difference visible directly: it plots model performance as a function of how much time has elapsed since training, so you can see the shape of the decay, not just one point on it.

Building the curve

Train a model once on data up to some cutoff date, then evaluate it separately on test windows at increasing distances from that cutoff, say, months 1, 2, 3, up to 12 after training ended, instead of pooling all post-training data into one aggregate score. Plotting the resulting performance metric (information coefficient, accuracy, or Sharpe ratio of a resulting strategy) against elapsed time since training gives the decay curve. In plain English: rather than asking "how good is this model on unseen data," ask "how good is this model at 1 month unseen, at 6 months unseen, at 12 months unseen," because those can be very different answers even though all three points are technically "out-of-sample."

Repeating this for several different training cutoffs and averaging the resulting curves smooths out the effect of any one lucky or unlucky historical period, giving a more reliable picture of the typical decay shape rather than one particular episode's.

Worked example: two signals with the same validation score, different decay

Signal A and Signal B both show an information coefficient of 0.05 in the first month after training. Tracing the full decay curve: Signal A's IC falls gradually, 0.05, 0.045, 0.04, 0.035, over the first four months, staying usefully positive for roughly a year. Signal B's IC falls off a cliff, 0.05, 0.02, 0.005, near zero, by month three, essentially useless from month four onward. If a researcher only checked the one-month score, both would look identical and equally worth deploying. The decay curves reveal that Signal A tolerates a slower, cheaper refit schedule (say, quarterly), while Signal B needs a much more frequent refit, monthly or even more often, to stay useful, or it needs to be abandoned if that refit cadence isn't operationally feasible.

What this means in practice

Decay curves directly inform two practical decisions: how often to refit (a curve that decays slowly tolerates infrequent, cheaper refits; a steep curve demands near-continuous retraining or should raise doubts about whether the signal is worth deploying at all), and how much to trust a signal's staying power when sizing a position or committing research effort to it. They also serve as an early-warning validation tool: if a live model's realized decay curve looks much steeper than the curve estimated during research, that's a sign either the original estimate was optimistic or the market relationship genuinely shifted.