Vendor Backfill and History Rewrites

It's tempting to think of historical market data as fixed — the closing price of a stock on a given day happened, and that's that. In practice, vendors revise their historical databases constantly. A vendor discovers a processing bug that misapplied a dividend three years ago and corrects the entire affected history in one update. A new data source becomes available and the vendor backfills years of a field that used to be blank. An exchange issues a late correction to a trade and the vendor updates its record to match. Each of these is reasonable and usually an improvement, but the net effect is that "the historical data" is not a single fixed thing — it's a moving target that gets silently rewritten underneath any research built on top of it.

Why this matters for reproducibility

If a backtest is re-run a year after it was first run, using "the same" historical database from the same vendor, it can produce different results purely because the vendor revised numbers in between — with no code change and no new data added, just corrected old data. This becomes a real problem when a strategy was originally validated on one version of history and is later re-validated, or compared against a newer version, without anyone realizing the ground truth itself moved. A result that looked robust against the old data might not survive against the corrected data, or vice versa, and without knowing which vintage of the database produced which result, there's no way to tell whether a discrepancy is a real finding or just a data revision.

A concrete case

A research team backtests a value strategy using historical fundamentals from a vendor, gets a strong result, and green-lights the strategy. Eighteen months later, an engineer re-runs the exact same backtest code to confirm the result before increasing capital allocation, pulling data from the same vendor connection — and gets a noticeably weaker Sharpe ratio. After investigation, the cause turns out to be a large batch of restated earnings figures the vendor backfilled in the interim, changing several years of the fundamental history the strategy depended on. Without a record of which data vintage produced the original result, the team can't easily tell whether the strategy's edge was real and has simply decayed, or was partly an artifact of since-corrected data errors.

What this means in practice

Serious research operations snapshot the data used for any important result — storing the exact vintage, not just a live connection to "the vendor's current database" — so a result can be reproduced against the data that actually produced it, and any later discrepancy can be traced to either a real change in the world or a data revision, rather than left ambiguous.