Incremental vs Full Feature Recomputation

Every night, a feature pipeline has to bring itself up to date with the day's new prices, trades, and filings. There are two fundamentally different ways to do that: recompute only what's new since yesterday and append it, or throw away yesterday's output entirely and rebuild the whole feature history from raw data. Both are used constantly in production quant systems, and picking the wrong one for a given feature causes two very different classes of expensive bugs.

Two strategies, one trade-off

Incremental recomputation takes just the new day's raw data, applies the feature logic, and appends the result to the existing table. It's fast — a pipeline with ten years of history only touches one new day's worth of rows, not ten years' worth — and it's the only realistic option once a table gets large enough that a full rebuild would take hours.

Full recomputation discards the existing feature table and recalculates every row, from the earliest date to today, using the current version of the feature logic. It's slow and expensive, but it has one property incremental updates lack: it's guaranteed to reflect the current code and data, with no leftover artifacts from an old, buggy version of the pipeline.

The danger with pure incremental updates is that they silently assume the past never needs to change. That assumption breaks constantly in finance: a vendor restates historical fundamentals, or someone fixes a bug in the feature formula itself. An incremental pipeline that only ever appends new rows keeps serving stale, wrong values for all the old rows, because nothing ever revisits them.

Worked example: a bug fix that incremental updates can't fix

A 20-day momentum feature was computed with an off-by-one error — a 19-day window — for the past two years, then the bug was fixed. If the pipeline only runs incrementally from here on, new days use the corrected 20-day window, but the entire two-year backfill still holds 19-day values. A model trained on that history learns from a feature that quietly changes definition partway through the dataset — producing a live model whose behavior doesn't match its backtest, with no error message pointing at the cause.

The fix requires a full recompute of the affected feature back to when the bug was introduced, followed by incremental updates from that point on. A common pattern: full recompute triggered explicitly whenever feature logic changes, cheap incremental updates whenever only new data arrived.

Think of that curve as "time to bring the table up to date" against "days of history involved": incremental recomputation stays flat and cheap day to day (like the curve's shallow tail), while a full recompute pays the whole steep climb every single time — worth it only when correctness, not speed, is what's at stake.

What this means in practice

The decision isn't made once for the whole pipeline — it's made per feature, and often per event. Routine new-data arrival is incremental almost everywhere; a change to feature code should trigger a scoped full recompute, tracked back to when the change was introduced. Teams that skip this end up with feature tables that are internally inconsistent across time with no code error to flag it — a model trained across that boundary inherits the inconsistency invisibly.