Feature Drift Detection in Production
Monitoring whether the statistical properties of a live model's input features are quietly changing over time, since a model trained on one distribution of inputs can degrade silently once live data starts looking different.
A machine learning model is trained on a snapshot of history, with its input features drawn from whatever distribution the market happened to produce during that training window. Live markets don't hold still — volatility regimes shift, new participants enter, market structure changes — so the distribution of live input features can gradually diverge from what the model was trained on, even while the model's code and parameters stay exactly the same. Feature drift detection is the practice of watching each input feature's statistical properties over time and flagging when they move meaningfully away from their training-period baseline, catching the problem before it shows up as degraded predictions.
The mechanics are usually simple: for each feature, track summary statistics like the mean, standard deviation, and the shape of its distribution over a rolling recent window, and compare them against the same statistics computed on the training data. A feature whose live mean has shifted by several standard deviations relative to training, or whose distribution has developed a new cluster of values the model never saw, is drifting — and a model making predictions from an input regime it was never trained on should not be trusted the same way it was when live data still resembled the training data.
A concrete example: a credit-risk model was trained using a bid-ask spread feature that, historically, rarely exceeded 20 basis points. After a major market structure change reduces liquidity in the traded universe, live spreads start regularly running at 60-80 basis points. The model's predictions haven't obviously broken — it still outputs numbers — but it is now extrapolating well outside the range of spread values it ever saw during training, and its outputs in that regime are unreliable in a way that ordinary performance monitoring might not catch immediately, since the model doesn't know it's guessing.
Feature drift detection is deliberately separate from tracking the model's actual prediction accuracy, because drift is often the earlier warning: features can visibly shift for weeks before enough labeled outcomes accumulate to show up as a measurable accuracy drop.
Feature drift detection tracks whether a live model's input features still resemble the data it was trained on, catching a model that has silently started extrapolating outside its training distribution before that shows up as degraded prediction accuracy.
Further reading
- Prado, Advances in Financial Machine Learning, ch. 8