Streaming Feature Computation

A feature like "20-day rolling average volume" is easy to compute in a batch job: load 20 days of history, average it, done. But a live trading system needs that feature updated every time a new trade prints, potentially thousands of times a second, and recomputing a 20-day average from scratch on every tick is wasteful and eventually too slow. Streaming feature computation solves this by maintaining the feature incrementally: keep just enough state to update the value cheaply as each new data point arrives, without re-reading the whole window.

Incremental update, not full recompute

For a simple moving average of window length $n$, the trick is to keep a running sum and drop the oldest point as each new one arrives:

$$S_t = S_{t-1} + x_t - x_{t-n}$$

In plain English: the new running total equals the old total, plus the newest value, minus the value that's now aged out of the window — an update that costs one addition and one subtraction regardless of how long the window is, instead of re-summing everything. Many common features have an equivalent incremental form: exponentially weighted moving averages update with a single blend of the old value and the new observation; running variance can be updated with Welford's algorithm; even more complex features like a rolling z-score can be built by combining a few incrementally maintained running statistics (sum, sum of squares, count).

Worked example

A market-making system tracks a 5-minute rolling average trade size to size its own quotes. Naively, on every new trade it could re-read the last 5 minutes of trade log and average it — at 200 trades per minute that's up to 1,000 rows re-averaged on every single new trade, an unnecessary $O(n)$ cost per update. Instead the system keeps a running sum and count for the current window in a small ring buffer that drops trades older than 5 minutes as new ones arrive. Each new trade updates the sum and count in constant time regardless of window size, well within the 1-millisecond quoting latency budget. Benchmarked, the naive recompute took 40 microseconds per update at peak load; the incremental version took under 0.1 microseconds.

What this means in practice

Any feature meant to feed a low-latency system should be designed with its incremental update rule in mind from the start, not bolted on after a batch version is built — some features (like a rolling median) have no cheap incremental form and either need an approximation or a bounded-cost data structure. Streaming and batch versions of the same feature must also be checked for numerical agreement periodically, since incremental algorithms can accumulate floating-point drift over long runs that a batch recompute won't show.