Monitoring Input Feature Distributions

A thermostat calibrated in a room where the sensor reads in Celsius will happily keep heating a house to a dangerous temperature if someone swaps in a sensor that reports the same numbers but in Fahrenheit — nothing about the thermostat's own logic changed, but the meaning of its input silently did. A trading model has the same vulnerability, many times over: a data vendor renormalizes a field, an exchange changes tick size, or the market simply enters a regime where a feature's typical range shifts. The model keeps running, producing confident-looking outputs, on inputs that no longer mean what they meant during training.

Comparing today's inputs to training's inputs

The standard tool is a divergence measure between the distribution of a feature at training time, $P$, and its distribution in the live data stream, $Q$. Population Stability Index (PSI) is a common, interpretable choice:

$\text{PSI} = \sum_{k} (Q_k - P_k) \ln\!\left(\frac{Q_k}{P_k}\right)$

In words: bucket the feature's values into bins (deciles are typical), compare what fraction of training data fell in each bin ($P_k$) to what fraction of today's live data falls in the same bin ($Q_k$), and sum up a weighted measure of how different those fractions are across all the bins. A PSI near 0 means the live distribution looks like training; a common rule of thumb treats PSI above roughly 0.25 as a major shift worth investigating, and above 0.1 as worth watching.

Worked example 1. A momentum feature was trained on data where 10% of stocks fell into the top decile bucket, by construction ($P_{10} = 0.10$). Six months into live trading, a market regime shift means 22% of stocks now fall into that same bucket range ($Q_{10} = 0.22$) — the feature's scale has effectively compressed, with far more stocks bunched at what used to be an extreme value. That single bucket's contribution to PSI is $(0.22 - 0.10) \times \ln(0.22/0.10) = 0.12 \times \ln(2.2) = 0.12 \times 0.79 \approx 0.095$. Summed across all ten deciles, a shift this pronounced in even one or two buckets is often enough to push total PSI past the 0.25 threshold on its own — a clear signal that the feature's meaning has drifted, even though the underlying calculation code hasn't changed at all.

Distinguishing a real shift from a bug

Worked example 2. Distribution monitoring also catches plain data bugs, which are far more common in practice than genuine regime shifts. Suppose an earnings-yield feature was trained with a mean of 0.06 and standard deviation 0.03 (a typical E/P ratio). One morning, live monitoring shows the feature's mean has jumped to 6.0 with standard deviation 3.0 — a factor of 100 too large. Computing PSI here is almost beside the point; a simple check on mean and standard deviation immediately flags that a vendor has switched a field from a decimal ratio (0.06) to a percentage (6.0%) without announcing it. Left unmonitored, a tree-based model might silently treat every stock as having an absurdly high earnings yield, potentially ranking every name identically at the extreme end of that feature's importance — a bug that a naive "did the model's output change a lot" check might not catch immediately, since the ranking damage depends on how the model's splits interact with the corrupted scale, but a feature-level distribution check catches on day one.

What this means in practice

Production ML pipelines typically compute PSI (or a similar measure, like KL divergence or a Kolmogorov-Smirnov test) for every input feature on a daily or weekly cadence, with automated alerts and, ideally, an automatic fallback to a simpler, more robust model or a reduced position size when a feature's PSI crosses threshold. This matters more, not less, as feature counts grow — a book running hundreds of engineered features has hundreds of independent ways for one of them to silently drift, and manually eyeballing each one doesn't scale past a handful.