Schema Drift and Field Definition Changes

Schema drift happens when a data field keeps the same name and column position over time, but the underlying definition, unit, or calculation methodology quietly changes underneath it. A "shares outstanding" field might switch from reporting basic shares to reporting diluted shares partway through a vendor's history; a "volume" column might switch from single-exchange to consolidated-tape volume after a vendor upgrade; an "earnings" field might change from as-reported to a restated basis without any flag in the data itself.

This is different from an obvious data error, because nothing looks broken — the field still populates every day with plausible-looking numbers, so nothing trips a simple null-check or range-check. The break only shows up as a subtle shift in the statistical relationship between that column and everything downstream: a feature built on it might show an unexplained regime change in its distribution, or a backtest might show performance flip at a date that has no economic meaning, and the real cause is a vendor changing their calculation rather than the market changing.

The main defense is holding a specific slice of history from each vendor to a known reference and re-checking it periodically, rather than trusting that a column that has "always worked" will keep meaning the same thing indefinitely — and keeping a change-log of vendor methodology notes, which most vendors publish but few users read.