Population Stability Index for Live Models

A model trained on last year's data assumes the world it sees live looks statistically similar to the world it learned from. When that stops being true — a feature's typical range shifts, or the mix of regimes changes — the model's predictions quietly become less reliable, often with no obvious error message. The Population Stability Index (PSI) is a simple way to catch this: bucket a feature (or the model's output score) into ranges, compare what fraction of observations falls in each bucket now versus during training, and combine the differences into one number.

$$PSI = \sum_i (p_i - q_i) \ln\!\left(\frac{p_i}{q_i}\right)$$

where $p_i$ is the fraction of current observations in bucket $i$ and $q_i$ is the fraction from the training period. Buckets where the current and training proportions match contribute almost nothing; buckets whose share has shifted a lot contribute more, and the log term punishes large relative shifts especially hard.

Worked example. Say a momentum feature was split into 5 buckets during training with proportions $q = (0.2, 0.2, 0.2, 0.2, 0.2)$. Live, the top bucket has grown to 0.35 and the bottom shrunk to 0.10, others roughly unchanged: $p = (0.10, 0.20, 0.20, 0.15, 0.35)$. The dominant terms come from the shifted buckets: $(0.35-0.20)\ln(0.35/0.20) \approx 0.084$ and $(0.10-0.20)\ln(0.10/0.20) \approx 0.069$, plus a smaller contribution from the 0.20 to 0.15 bucket; summing all five gives $PSI \approx 0.17$. The common rule of thumb is PSI below 0.1 is stable, 0.1–0.25 is worth watching, above 0.25 signals meaningful drift — so 0.17 here would flag the feature for a closer look, not an automatic shutdown.