Training-Serving Skew

A quant builds an alpha model, backtests it carefully, and it looks strong. It goes live and the returns are mediocre, sometimes for months before anyone can say why. Nobody changed the model. What often happened instead is quieter and harder to see: the feature the model was trained on and the feature it is fed in production are not actually the same calculation, even though they have the same name. That mismatch is training-serving skew, and it is one of the most common ways a good model fails for a bad reason.

Two pipelines pretending to be one

In research, features are usually computed by a batch job running over historical data sitting in a dataframe: 20-day rolling volatility, sector-relative momentum, an earnings-surprise flag. In production, the same feature has to be computed on the fly from a live data feed, often in a different language, by a different team, under a latency budget. The research pipeline might use a pandas rolling window that looks backward and forward by accident (a common bug), while the serving pipeline can only ever look backward. The research pipeline might fill missing values with the full-sample median; the serving pipeline has no future data to compute a full-sample median from and has to use a running one instead. Each of these is a small, reasonable-looking implementation choice made independently, and each one shifts the distribution of the feature the model actually sees at inference time away from what it saw in training.

Worked example: a volume feature computed two ways

Suppose the feature is "today's volume as a multiple of the 20-day average volume." In research, the 20-day average is computed with df['volume'].rolling(20).mean() over a clean, corporate-action-adjusted history, and a stock that traded 2 million shares against a 20-day average of 1 million gets a feature value of 2.0.

In production, the serving system computes the same average from a streaming feed that has not yet applied the split adjustment for a stock that split 2-for-1 overnight. The "20-day average" it computes is contaminated by 10 days of pre-split volume figures that are roughly half the post-split scale. If the pre-split average volume was genuinely around 1 million shares (in pre-split share counts, so ~500,000 in post-split terms), the serving pipeline's blended 20-day average comes out near 750,000. Today's actual volume of 2 million shares divided by 750,000 gives a feature value of about 2.67 instead of 2.0 — a 33% distortion driven entirely by a corporate-action handling difference between the two pipelines, not by any real change in the stock's behavior. The model, trained to associate a reading of 2.0 with a certain expected move, is now looking at a reading of 2.67 for the exact same underlying event, and its prediction is wrong for a reason that has nothing to do with the market.

What this means in practice

The most reliable fix is to make the training and serving pipelines share the same code, not just the same specification. If a rolling average is computed once in a shared library that both the offline feature-generation job and the live inference service import, the two paths cannot silently diverge. Where that is impractical — say the serving system is in C++ for latency and research is in Python — the discipline shifts to unit tests that feed identical raw inputs through both pipelines and assert the outputs match to a tight tolerance, run continuously as part of the deployment pipeline rather than checked once at launch. A second useful habit is logging the actual feature values seen at inference time and periodically diffing their distribution against the training-set distribution for the same feature; a persistent, unexplained shift is often skew rather than genuine market drift.

Practice

1. A research pipeline forward-fills missing prices; the serving pipeline, which has no future data, has to fill only with the last known price. Explain why this could look identical on a backtest that omits any period with missing data, and only surface once real gaps occur live.
2. List two ways to verify that a feature computed in a Python research notebook and a feature computed in a production Java service are numerically identical.