Batching and Vectorised Inference
Restructuring model inference code to operate on arrays of inputs at once using matrix operations, instead of looping over inputs one at a time in a slow interpreted language.
Adding up two lists of a million numbers with a Python for loop works, but it's slow, because every addition pays the overhead of the interpreter checking types and dispatching the operation. Handing the same lists to NumPy as arrays and calling one addition lets compiled code do all million additions in a single tight loop with none of that per-element overhead — the same math, computed far faster, purely because of how the work is organized.
Vectorised inference applies this to running a trained model: instead of calling model.predict(x) once per row inside a loop, the full batch of rows is stacked into a single array and passed through as one matrix multiplication. The underlying linear algebra libraries are built for exactly this kind of bulk, uniform-shaped computation, and looping in Python defeats that by forcing thousands of tiny, separately-dispatched calls instead of one large one.
Take a logistic regression model scoring 100,000 rows with 20 features. Looping row by row might take on the order of 10 seconds due to per-call overhead. Stacking all 100,000 rows into one array and computing the single matrix-vector product typically finishes in well under 100 milliseconds — a 100x-plus speedup for identical arithmetic, purely from batching the operation.
Stacking many inputs into a single array and running one batched matrix operation lets hardware-optimized linear algebra do the work, avoiding the per-call interpreter overhead that makes row-by-row Python loops over a model dramatically slower for the exact same computation.
Further reading
- VanderPlas, Python Data Science Handbook, ch. on vectorization