SGD Noise and Implicit Regularization

Explicit regularization is a deliberate penalty you add to the loss — an L2 term, dropout, early stopping — something you chose on purpose to fight overfitting. But networks trained with plain mini-batch stochastic gradient descent, no explicit penalty at all, still often generalise better than the same network trained with exact full-batch gradient descent on the identical data. Something about the procedure itself, not any penalty you bolted on, is doing regularising work. That something is the noise mini-batching introduces.

The analogy: a hiker who can't see the whole map

A full-batch optimiser is a hiker with a perfect map of the entire valley, walking the single steepest path down to whatever the lowest point happens to be — including a narrow, easily-missed crack if that's genuinely the deepest spot. A mini-batch optimiser is the same hiker in dense fog, only ever seeing the terrain immediately underfoot from a small sample of the ground. Each step is a decent local guess, but never a precise reading of the true steepest direction. That fog has a side effect: the hiker physically cannot thread the needle into a narrow crack, because doing so would require a precision the noisy readings don't have. The fog naturally steers the hiker toward wide, forgiving basins that are easy to stumble into and hard to stumble out of — exactly the flat regions that generalise well.

Where the noise comes from

Full-batch gradient descent computes the exact gradient over all $N$ training points. Mini-batch SGD estimates it from a random subset of size $B$:

$$g_{\text{batch}} = \frac{1}{N}\sum_{i=1}^{N} \nabla \ell_i(\theta), \qquad g_{\text{SGD}} = \frac{1}{B}\sum_{i \in \mathcal{B}} \nabla \ell_i(\theta)$$

In words: the true gradient averages every training example's contribution; the SGD estimate averages only a random handful of them, and by how much it misses the true average shrinks as the batch gets bigger. The gap between the two, $g_{\text{SGD}} - g_{\text{batch}}$, behaves like injected noise with variance that scales roughly as $1/B$ — small batches inject more noise, large batches inject less, and a full batch injects none. That noise doesn't cancel to zero at each step; it accumulates into a persistent random wobble around the descent path, and the wobble is large exactly where the loss surface curves sharply, which is precisely the signature of a narrow minimum. A ball rolling downhill with a persistent random jitter cannot settle at rest in a spot that punishes small deviations — it gets kicked back out.

Worked example 1: batch size versus effective noise, by hand

Take a toy loss surface where the gradient noise variance is $\sigma^2 = 4$ per example. With batch size $B = 10$, the variance of the batch gradient estimate is $\sigma^2 / B = 4/10 = 0.4$, so its standard deviation (the typical wobble size) is $\sqrt{0.4} \approx 0.63$. With batch size $B = 1000$, variance is $4/1000 = 0.004$, standard deviation $\approx 0.063$ — the wobble shrinks by roughly a factor of ten when the batch grows by a factor of one hundred, because noise scales with the square root of the batch size, not linearly. This is the direct mechanical reason large-batch training reaches sharper, more precisely-targeted minima: the optimiser simply has a much steadier hand.

Worked example 2: same architecture, two training runs

A network trained with batch size 32 and a network trained with batch size 8192 both reach training accuracy of 99.9% on the same dataset — full-batch-like precision on the training set either way. Post-training sharpness measurements (loss increase after a fixed weight perturbation, as in Loss Landscape Geometry and Sharpness) come out to $0.02$ for the small-batch run and $0.45$ for the large-batch run. On a held-out validation split, the small-batch model's accuracy drops to 97.5%, a gap of 2.4 points; the large-batch model's accuracy drops to 91.0%, a gap of 8.9 points — roughly three and a half times worse degradation, from a training run that looked identical on paper. Nothing about the loss function changed between the two runs; only the batch size, and therefore the noise level, did.

What this means in practice

This is why "just use the biggest batch your GPU memory allows" is not a free lunch — it removes a regularising force you were relying on without realising it. The common fixes acknowledge this directly: scale the learning rate up roughly in proportion to batch size to partially restore an equivalent noise level, add an explicit warm-up period, or reintroduce noise deliberately through other channels (dropout, data augmentation, label smoothing) to compensate for a large batch's precision. On financial datasets, where the temptation is often to use large batches purely for training-speed reasons on GPU clusters, this trade-off deserves explicit attention: a model that trains twice as fast but generalises measurably worse out-of-sample has not actually saved you anything.

The same logic explains why some practitioners deliberately keep batch sizes modest even when a larger batch would speed up wall-clock training time, treating the resulting noise as a regulariser worth its computational cost rather than a nuisance to be engineered away. It's also a useful diagnostic in reverse: if a model trained with a large batch and a carefully tuned learning rate still generalises noticeably worse than a small-batch baseline, sharpness of the resulting minimum, not an error in the learning-rate schedule, is a natural first suspect.

Try a large step size in the explorer and watch the path overshoot and oscillate rather than gliding smoothly to the bottom — SGD's mini-batch noise produces a related, persistent version of that oscillation, which is what keeps it out of the sharpest, easily-overfit minima.