Streaming Feature Computation

A feature like "20-day rolling average volume" is easy to compute in a batch job: load 20 days of history, average it, done. But a live trading system needs that feature updated every time a new trade prints, potentially thousands of times a second, and recomputing a 20-day average from scratch on every tick is wasteful and eventually too slow. Streaming feature computation solves this by maintaining the feature incrementally: keep just enough state to update the value cheaply as each new data point arrives, without re-reading the whole window.

Incremental update, not full recompute

For a simple moving average of window length $n$, the trick is to keep a running sum and drop the oldest point as each new one arrives:

In plain English: the new running total equals the old total, plus the newest value, minus the value that's now aged out of the window, an update that costs one addition and one subtraction regardless of how long the window is, instead of re-summing everything. Many common features have an equivalent incremental form: exponentially weighted moving averages update with a single blend of the old value and the new observation; running variance can be updated with Welford's algorithm; even more complex features like a rolling z-score can be built by combining a few incrementally maintained running statistics (sum, sum of squares, count).

Worked example

A market-making system tracks a 5-minute rolling average trade size to size its own quotes. Naively, on every new trade it could re-read the last 5 minutes of trade log and average it, at 200 trades per minute that's up to 1,000 rows re-averaged on every single new trade, an unnecessary $O(n)$ cost per update. Instead the system keeps a running sum and count for the current window in a small ring buffer that drops trades older than 5 minutes as new ones arrive. Each new trade updates the sum and count in constant time regardless of window size, well within the 1-millisecond quoting latency budget. Benchmarked, the naive recompute took 40 microseconds per update at peak load; the incremental version took under 0.1 microseconds.

What this means in practice

Any feature meant to feed a low-latency system should be designed with its incremental update rule in mind from the start, not bolted on after a batch version is built, some features (like a rolling median) have no cheap incremental form and either need an approximation or a bounded-cost data structure. Streaming and batch versions of the same feature must also be checked for numerical agreement periodically, since incremental algorithms can accumulate floating-point drift over long runs that a batch recompute won't show.