Message Queues and Log-Based Streaming

If a feed handler writes market data directly into a strategy's process and the strategy crashes, the feed handler either has to buffer everything in memory until the strategy comes back or lose data entirely. A message queue sits between the two: the feed handler publishes events to the queue and moves on, and the strategy reads from the queue whenever it's ready, so a crash in one component doesn't directly take down the other. This decoupling — producers and consumers that don't need to be running, or fast, at the same moment — is the basic reason queues exist in any pipeline with more than one moving part.

Traditional queues (think of a literal line at a counter) typically delete a message once a consumer has read it. That's fine for a single consumer that just needs to process each event once, but it breaks down the moment a second consumer wants to read the same stream — the risk system and the research logger both want every trade event, but the risk system reading first would delete the message before the logger ever saw it. Log-based streaming solves this by treating the stream as an append-only, ordered record that consumers read from without removing anything; each consumer just tracks its own position (an "offset") in the log and can replay from any earlier point. Multiple consumers can read the same log independently, and a consumer that crashes and restarts can pick up exactly where it left off instead of losing whatever arrived while it was down.

The log-based model also makes replay a first-class feature rather than an afterthought. Because messages aren't deleted on read, a new consumer added months later — say, a new monitoring dashboard — can start from the beginning of the log (or any offset) and reconstruct the full history of events, which is invaluable both for debugging ("replay exactly what the system saw before it made this bad decision") and for backfilling a new downstream system without needing a separate historical data pipeline.

Worked example: one stream, two independent readers

A trade event stream is published once by the exchange feed handler. The risk system consumes it at offset 500, checking position limits after every trade. Separately, an overnight archiver consumes the same stream but is currently behind, at offset 200, because it batches writes to disk every few minutes. With a log-based design, both consumers read independently from the same underlying data — the risk system being 300 messages ahead doesn't delete anything the archiver still needs, and if the archiver process restarts after a crash, it resumes from offset 200 exactly, replaying nothing it already wrote and losing nothing that arrived in between.

What this means in practice

Log-based streaming is the standard choice whenever more than one consumer needs the same event stream, or whenever replay-ability (for debugging, backfilling, or auditing) matters — which describes most market data and trade-event infrastructure at a quant firm. Simpler point-to-point queues are still reasonable for single-consumer, fire-and-forget tasks like a one-off background job where nothing ever needs to reread the same message twice.