Designing the Research Data Layer

Every researcher on a desk eventually writes their own version of the same code: load prices, join to fundamentals, handle a handful of missing tickers, adjust for splits. Multiply that by ten researchers and you get ten slightly different, silently inconsistent versions of "the data," each with its own small bugs. The research data layer is the shared infrastructure that does this loading and cleaning once, in one place, so every researcher's model reads from the same trusted source instead of reinventing it.

What belongs in the layer

A well-designed data layer typically separates into stages: raw ingestion (vendor files, untouched), a cleaning and standardization layer (consistent identifiers, consistent units, documented handling of nulls), and a query interface researchers actually call from their own code, which hides the storage format and the joins behind a simple, stable function signature.

Worked example

Before a data layer existed, a nine-person research team discovered that four different notebooks handled a stock's mid-year ticker change four different ways — one kept both old and new tickers as separate entities, two silently dropped the pre-change history, and one manually patched it. Results from those four notebooks disagreed on a shared backtest by amounts that had nothing to do with actual research disagreement. After building a shared entity-resolution table into the data layer's cleaning stage, all four notebooks began pulling the same, single, agreed-upon history for that name, and a follow-up comparison of their results converged to within rounding error on the parts of the analysis that used only that shared data.

What this means in practice

Invest in the data layer before the team's headcount grows, not after — retrofitting shared plumbing under ten already-diverged research codebases is far more expensive than building it under two.