Batch vs Streaming Pipelines

A batch pipeline runs on a schedule — every night at 2 AM, process the day's trades, recompute risk numbers, rebuild a research dataset — treating each run as a self-contained job over a fixed chunk of data. A streaming pipeline processes each new piece of data as it arrives, continuously, with no notion of "the job is done" — a live feed handler updating positions tick by tick never finishes running. Both are legitimate ways to move data from a source to a usable form, and the right choice depends almost entirely on how fresh the output needs to be and how the downstream consumer wants to use it.

Batch's strength is simplicity and correctness at rest: a nightly job that reprocesses a full day's data can afford to wait for every source to finish updating, retry on failure without racing a live clock, and produce a single, complete, internally consistent snapshot. This is exactly what most research and backtesting infrastructure wants — a clean daily dataset that doesn't change underneath you while you're querying it. Its weakness is latency: the output is only ever as fresh as the last run, so a batch job that runs nightly means anything that happened since last night's run isn't reflected yet, which is fine for research but useless for anything that needs to react during the trading day.

Streaming's strength is exactly that freshness: a live risk system or a market-making strategy needs to know about a trade the moment it happens, not tomorrow morning. Its cost is real complexity — a streaming pipeline has to handle out-of-order and late-arriving events, define what "the window is complete" means when data can still trickle in, and keep running reliably for weeks or months without the natural reset point a batch job gets for free every night. Streaming systems are also harder to debug: reproducing a batch job's output means re-running it on the same input, but reproducing a streaming bug means recreating the exact timing and ordering of events that triggered it.

Worked example: end-of-day PnL vs live risk

An end-of-day PnL report that feeds tomorrow's research is naturally batch: it reads a full day of completed trades once, after the market closes and all late corrections have settled, and produces one authoritative number per strategy. Running it as a stream — updating PnL continuously through the day — would add real complexity for no benefit, since nobody needs an intraday PnL number to be reproducible to the tick for a report that's only read the next morning. A live risk limit check, by contrast, has to be streaming: if a position breaches a limit at 11:47 AM, a system that only checks limits at 2 AM that night has already let the risk sit unmanaged for hours.

What this means in practice

Most quant infrastructures run both, purpose-built for different jobs: streaming for anything that needs same-day or sub-second reaction (feed handlers, live risk, order management), batch for anything that wants a clean, complete, reproducible snapshot (research datasets, backtests, end-of-day reporting). Trying to force one pattern to do both jobs — a nightly batch job pretending to be real-time, or a streaming system used where a simple daily snapshot would do — usually adds cost without adding value.