Latency vs Throughput

A highway can move a huge number of cars per hour and still make every single driver's individual trip slow, if it's congested. It can also move very few cars per hour while each one travels at top speed with an empty road ahead. These are two different questions — how long does one car's trip take, and how many cars get through per hour — and a change that helps one can actively hurt the other. Trading systems face the identical tension, and confusing the two is a common, costly design mistake.

The idea: one item's time versus items per second

Latency is the time from when a single request starts to when it finishes — for a trading system, commonly the "tick-to-trade" time: from a market-data packet arriving to an order leaving in response. Throughput is the rate at which the system completes work — messages processed per second, orders sent per second. They are related but not interchangeable: a system processing messages one at a time with zero queuing has latency equal to the processing time of one message, and throughput equal to 1 / (that processing time). But the moment a system does anything to increase throughput by working on multiple items at once — batching, pipelining, using multiple cores — the relationship breaks, because now an individual item can sit waiting for others before it's handled.

Batching is the clearest example of the tradeoff. Waiting to collect 100 market-data updates before processing them together (to amortize some fixed per-batch cost, e.g. reducing System-call overhead) raises throughput — you do less fixed-cost work per message on average — but it raises the latency of the first message in the batch, because that message now sits waiting for 99 others to arrive before anything happens to it.

Worked example: single-item processing versus batching

Suppose processing one market-data message, on its own, takes 2 microseconds of actual CPU work, but there's also a fixed 10-microsecond overhead per batch (e.g. a lock acquisition, a syscall, a queue dequeue operation) regardless of how many messages are in it.

No batching (batch size 1): each message pays the full 10 μs fixed cost plus 2 μs of work = 12 μs latency per message. Throughput: 1 message / 12 μs = about 83,000 messages/second.

Batch size 100: the fixed 10 μs cost is paid once for the whole batch of 100, so average per-message overhead drops to 10/100 = 0.1 μs, plus the 2 μs of actual work = 2.1 μs average processing cost per message. Throughput: roughly 1 / 2.1 μs ≈ 476,000 messages/second — a 5.7x improvement.

But trace what happened to the first message in that batch of 100: it doesn't get processed the instant it arrives. It waits for the other 99 messages to accumulate before the batch fires. If messages arrive steadily, that wait alone can be tens or hundreds of microseconds, dwarfing the 2.1 μs average processing cost. The message that determines your worst-case latency paid a huge price for a throughput number that looks great on average. This is exactly why an HFT tick-to-trade path almost never batches, even though batching would make the system's aggregate numbers look better.

# Per-message cost is fixed regardless of batching; only the OVERHEAD amortizes
FIXED_OVERHEAD_US = 10
WORK_PER_MSG_US = 2

def avg_latency_per_msg(batch_size):
    return FIXED_OVERHEAD_US / batch_size + WORK_PER_MSG_US

for b in [1, 10, 100]:
    print(b, avg_latency_per_msg(b), "us/msg average")
# 1   12.0 us/msg average
# 10   3.0 us/msg average
# 100  2.1 us/msg average -- but the FIRST message in the batch waited far longer

batched (size 4, shown) wait for batch to fill process all 4 together high throughput, but first msg waited for the whole batch to arrive

Little's Law: the relationship, made precise

The three quantities are linked by Little's Law: $L = \lambda W$, where $L$ is the average number of items in a system (in flight), $\lambda$ (lambda) is the arrival rate (throughput, items per second), and $W$ is the average time an item spends in the system (latency). It says, in words: the number of things queued up equals the rate they arrive times how long each one stays. Push throughput up while latency per item stays fixed, and the number of in-flight items rises proportionally — which is exactly why a system under heavy load, if it can't actually process faster, ends up with both worse latency (queueing) and a throughput ceiling it can't exceed no matter how much load you throw at it.

Where this shows up

In a quant-dev interview, this distinction is a standard systems-design probe: "would you rather have 1 μs average latency with occasional 1 ms spikes, or a steady 5 μs" tests whether a candidate understands that tail latency, not average, is usually what matters for trading, and that the two metrics genuinely optimize in different directions. In production, market-data and order-entry paths are engineered almost exclusively for latency (specifically worst-case, or tail latency) even at some cost to throughput — batching, buffering, and queuing are avoided on purpose — while systems like end-of-day risk aggregation or historical data ingestion are engineered for throughput, where individual-item latency is irrelevant and batching is free performance.