Data Versioning and Lineage

A backtest result depends on more than just the strategy code — it depends on the exact data that fed it, and that data changes over time as vendors revise history and internal pipelines get updated. If a researcher can only say "I used the standard price database" without specifying which version of it, the result can't reliably be reproduced later, because "the standard price database" today might not be the same numbers it was six months ago. Data versioning solves this by treating each meaningful version of a dataset as a distinct, permanently identifiable snapshot — much like source code is versioned with git — so any result can be tied back to the exact data that produced it.

Lineage: knowing where a number came from

Versioning answers "which snapshot was this?" Lineage answers the related question "where did this specific number come from, and what was done to it?" A cleaned, adjusted price series in a research database might trace back through several transformation steps — raw vendor feed, bad-tick filter, corporate-action adjustment, resampling to daily bars — and lineage tracking records that chain, so if a number looks wrong, an engineer can walk backward through each step to find where the problem was introduced, rather than treating the pipeline as an opaque box.

A concrete case

A strategy's backtested Sharpe ratio drops noticeably after a routine data refresh, with no code changes. Without versioning, this is a confusing, hard-to-debug mystery — is the strategy's edge genuinely fading, or did something change in the data? With versioning, the researcher can pull up the exact prior data snapshot, re-run the identical backtest against it, and confirm the old result reproduces exactly — isolating the change to the data refresh. Lineage then shows which upstream transformation changed: perhaps the bad-tick filter's threshold was recently retuned and now flags a wider set of "genuine" volatile moves as errors, quietly reshaping the input data the strategy actually saw.

What this means in practice

Practically, this means storing data as immutable, timestamped snapshots rather than always querying "the current version" of a live database, and logging enough metadata about each transformation step that the full chain from raw feed to final input can be reconstructed. This adds real infrastructure overhead, which is why many small research operations skip it — but the cost shows up later, as untraceable discrepancies between old and new results that eat far more time to debug than the versioning would have taken to set up.