Batch Size and Generalization

Batch size — how many training examples get averaged into a single gradient estimate before the network's weights update — looks like a pure engineering knob for speed: bigger batches use hardware more efficiently and finish an epoch faster. But it also changes where in the loss landscape training ends up, which affects how well the model performs on data it hasn't seen.

Small batches produce noisier gradient estimates, and that noise acts like a mild random search that tends to push training away from narrow, sharp minima and toward wider, flatter ones. Flat minima generalize better because a small shift between the training and test loss surfaces still leaves you near the bottom; a sharp minimum can turn a tiny shift into a big loss increase. Large-batch training, by averaging over many examples, produces a smoother but more precise gradient that can settle into these sharp minima and generalize noticeably worse, even when training loss looks identical.

For example, empirical studies training the same architecture with batch size 256 versus batch size 8192 have found the large-batch model reaching similar or lower training loss but several percentage points worse test accuracy — the classic large-batch generalization gap.

In practice this is why practitioners pair large batches with techniques like learning-rate warmup, larger learning rates scaled to batch size, or explicit sharpness-aware optimizers, rather than simply maximizing batch size for throughput.