Data Drift vs Concept Drift

A model's live performance degrades. Before doing anything about it, the first question is why, and there are two structurally different answers that call for different responses. Data drift means the distribution of inputs the model sees has changed — feature values are shifting even though the underlying relationship the model learned is still valid. Concept drift means the relationship itself has changed — the same input now maps to a different correct output than it used to, even if the inputs' distribution looks unchanged.

Telling them apart

Data drift shows up as a change in $P(X)$, the distribution of the features, and can be detected without any labels at all — just by comparing the distribution of live inputs to the distribution the model was trained on, using tools like the population stability index or a maximum mean discrepancy test. Concept drift shows up as a change in $P(Y \mid X)$, the relationship between features and the correct label, and is fundamentally harder to detect quickly because it requires labels — you need to eventually learn the true outcome to notice the mapping has shifted, and in trading that outcome (realized return, whether a trade was profitable) often only becomes known after a delay.

A model can face pure data drift with no concept drift (inputs look different but old relationships still hold — often harmless, sometimes even fine to ignore), pure concept drift with no data drift (inputs look the same but now mean something different — the most dangerous and hardest to catch case, since input monitoring alone won't flag it), or both at once.

Worked example

A credit-spread prediction model was trained on data where investment-grade bond spreads ranged mostly between 80 and 150 basis points. In a new market environment, spreads for the same universe now range between 200 and 400 basis points — a clear shift in $P(X)$, detected immediately via a population stability index that jumps from 0.05 (stable) to 0.31 (major shift). That alone is data drift: it says nothing about whether the model's learned relationship between issuer leverage and spread is still correct. Separately, six months later, a tax-treatment change alters how the market prices leverage risk — a bond with a given leverage ratio now trades tighter than the model, trained on the old relationship, expects. Spread levels look unremarkable through this period, so the population stability index on inputs stays near 0.04, showing no data drift — but as realized spreads come in, prediction error creeps from an average of 8 basis points up to 22, revealing concept drift that input monitoring alone would have missed entirely.

What this means in practice

Input-distribution monitoring is necessary but not sufficient: it catches data drift cheaply and quickly, but a model can pass every input-drift check while quietly getting the relationship wrong, which is why performance-estimation methods that work without immediate labels are the complementary tool needed to catch concept drift before too much damage accrues.