Data Lineage and Provenance

A backtest result depends on a chain of data: raw prices, corporate action adjustments, a universe filter, a feature calculation, a model, and finally a P&L number. Data lineage is the record of that whole chain — which raw source fed which transformation, in what order, at what time — so that if a number looks wrong, someone can trace backward to find exactly where it came from, and if a source turns out to be wrong, someone can trace forward to find every result it touched.

Without lineage, a quant discovering that a vendor's dividend field was silently backfilled with a wrong value has no way to know which of the firm's hundred running backtests used that field, whether the error affected live trading decisions, or whether it's safe to keep trusting other fields from the same vendor. With lineage tracked — even something as simple as tagging every derived dataset with its source dataset's version and timestamp — that same discovery becomes a fifteen-minute query instead of a firm-wide fire drill.

Provenance also matters for research integrity on its own: a result that can't be traced back to its raw inputs can't be independently reproduced or audited, which is exactly the property a golden dataset or regression test is designed to protect.