Batch vs Online Inference

Once a model is trained, it has to actually produce predictions for someone to use — and there are two fundamentally different ways to organize that. Batch inference collects a large pile of inputs and runs the model over all of them together, on a schedule, producing a big pile of predictions that get stored for later use. Online inference runs the model on a single input the moment it arrives, returning a prediction immediately, one request at a time. The same trained model, the same weights, can be served either way — the difference is entirely in how requests are grouped and when the answer is needed, and that choice has large, very practical consequences.

The analogy: a mail sorting facility versus a live phone line

A mail sorting facility processes an enormous stack of letters overnight in one continuous run, and nobody is waiting for any one specific letter — the whole batch finishes together, hours later. A live customer-service phone line is the opposite: each caller needs an answer right now, one call at a time, and the system has to be ready to respond the instant a call comes in, even if that means sitting idle between calls. Both handle volume, but demand different infrastructure: the mail facility is optimized for total throughput; the phone line for how fast any single caller gets an answer.

What actually differs

Batch inference processes $N$ inputs together, letting the model exploit hardware parallelism across the whole batch — a GPU processing 10,000 inputs together is far more efficient per-input than processing 10,000 inputs one at a time, since much of a GPU's compute is wasted when it's fed only a single example:

$$\text{throughput} \propto \text{batch size (up to hardware limits)}, \qquad \text{latency per individual result} = \text{time for the whole batch}$$

In words: batching raises overall throughput (predictions produced per second, averaged over the batch) by keeping hardware busy, but the cost is that any single input's result isn't available until the entire batch finishes — a bad trade if someone is waiting on one specific answer right now. Online inference flips this: each request gets a fast, individual response (low latency per request), but hardware utilization per request is much lower, since the model runs on tiny amounts of data (often size 1) at a time, well below what would keep the hardware fully busy.

Worked example 1: batch throughput advantage

A GPU can process a batch of 256 inputs through a model in 50ms total. Per-input time: $50/256 \approx 0.195$ms. The same GPU processing inputs one at a time might take, say, 5ms per single input due to fixed per-call overhead not being amortized across a batch — 25 times slower per input than the batched version, purely from underusing the hardware's parallel capacity on a batch size of 1.

Worked example 2: online inference's latency advantage

A recommendation system needs to respond to a user's page load within 100ms or the page feels sluggish. If it used a nightly batch job instead, that single user's specific request would have to wait until the next scheduled batch run — potentially hours away — which is unacceptable for anything the user is actively waiting on. Online inference here isn't chosen for efficiency; it's the only option that can meet a hard per-request latency requirement, even though it uses hardware less efficiently on a per-prediction basis than a large overnight batch would.

What this means in practice

The choice is driven by use case, not by which is "better": fraud scoring at a card swipe, live search ranking, and chatbot responses all require online inference because one specific answer is needed immediately; overnight risk scoring across a loan portfolio, or nightly demand forecasts per SKU, fit batch inference because nobody needs any single result before the whole job finishes. Some systems use both: a batch job precomputes and caches predictions for common cases, while online inference handles requests that fall outside what was precomputed.

Practice

1. A model takes 60ms to process a batch of 300 inputs together, or 4ms per input one at a time. What is the per-input time in the batched case, and how many times faster is it than one-at-a-time?
2. Name one production use case that clearly requires online inference and explain why batch wouldn't work.
3. Why can a system reasonably use both batch and online inference for different parts of the same product?