Incremental Recomputation When Data Updates

A ten-year daily factor pipeline finishes computing today, and tomorrow one new day of prices arrives. The naive approach reruns the entire ten-year history from day one to compute tomorrow's single new row — repeating 2,500 days of work to get 1 new day of output. That's not just slow; it's wasteful in a way that gets worse every single day, since the pipeline keeps growing while the genuinely new information stays a constant, tiny slice. Incremental recomputation asks a sharper question: given what's already been computed, what is the minimum additional work needed to fold in the new data?

Recognizing what actually needs to change

Some computations are naturally incremental: a rolling 60-day average only needs the newest day's value and the value dropping out of the window, not a full 60-day resum every time. Others are structurally not: a cross-sectional rank on a given day depends on every name in that day's universe, so if a new name gets added retroactively to a historical date, every rank on that date has to be redone — there's no shortcut. The dividing line is whether an old output can be updated using only the old output plus the new increment, or whether it fundamentally depends on the whole dataset being reconsidered together. Knowing which category a step falls into is the entire design decision; treating a non-incremental step as if it were incremental produces answers that quietly stop matching what a full recompute would give.

Worked example: a rolling-average feature two ways

A 60-day rolling volatility feature for one stock, recomputed the naive way, resums 60 values of squared returns every single day even though 59 of those 60 values were already summed yesterday — that's 60 units of work per day, forever. Incrementally, you keep yesterday's sum, subtract the day that just fell out of the window, add the day that just entered, and divide by 60 — 2 units of work per day instead of 60, a 30x reduction that only grows as the window lengthens. Over a year, that's the difference between roughly 15,000 units of wasted recomputation and roughly 500 units of genuinely new work, for identical output.

What this means in practice

Incremental pipelines are worth building once a dataset update happens routinely — daily prices, weekly fundamentals — and the cost of a full recompute is large enough to matter, but they add real complexity: bugs in incremental logic are notoriously harder to catch than bugs in a fresh full computation, because a subtly wrong incremental update can drift from the true answer for months before anyone notices, while a full recompute is at least consistently right or consistently wrong. Most research teams keep a full-recompute path around even after building an incremental one, and periodically run it in parallel to check the two agree — treating the incremental version purely as a speed optimization, never as the sole source of truth.