Inference Batching and Throughput

A bus that waits to fill up before leaving moves more passengers per driver-hour than a taxi that leaves the instant one rider gets in, but each rider on the bus waits longer to depart. Serving predictions from a trained model faces the same trade-off: a GPU running one input at a time leaves most of its parallel compute idle, because the same weights are being multiplied against a tiny amount of data.

Batching collects incoming requests over a short window — a few milliseconds, or until a batch fills to a target size — and runs them through the model together as one matrix operation instead of many small ones. This uses far more of the hardware's parallel capacity per unit of work, but the first request in a batch must wait for the whole batch to fill before any of them return.

Take a service with a 5ms fixed per-batch compute cost and a 10ms max wait to accumulate a batch. A lone request under light traffic waits up to 10ms for the window to close before its 5ms of compute even starts — 15ms total instead of 5ms unbatched. Under heavy traffic, batches fill almost instantly, so the wait collapses toward zero while the system still serves far more requests per second for roughly the same 5ms of per-batch compute.