Gradient Noise Scale and Critical Batch Size

Training with a bigger batch always reduces the noise in a gradient estimate, but that does not mean bigger is always better — past a certain point, halving the noise stops translating into meaningfully faster convergence, because the remaining noise was never the bottleneck to begin with. The gradient noise scale is a single number that predicts roughly where that turning point sits, before you spend the compute finding out the hard way.

The analogy: how big a poll actually needs to be

Polling 50 people to estimate an election result gives a noisy read; polling 500 tightens it up substantially. But polling 50,000 people barely improves on polling 5,000 in any way that changes a close decision — past some sample size, the population's genuine variability, not sample size, dominates how precise you can get, and more respondents mostly cost money without buying much more certainty. Batch size in training plays the same role as sample size in a poll: a bigger batch gives a less noisy estimate of the true full-dataset gradient, but only up to the point where the underlying "how much do individual gradients disagree with each other" sets a floor that more data per step can't push through efficiently.

The gradient noise scale, defined

The gradient noise scale $B_{\text{noise}}$ compares how much individual training examples' gradients disagree (the trace of their covariance, $\text{tr}(\Sigma)$) to how large the true underlying gradient actually is ($|G|^2$):

$$B_{\text{noise}} \approx \frac{\text{tr}(\Sigma)}{|G|^2}$$

In words: the numerator measures how noisy a single example's gradient typically is relative to the average, and the denominator measures how big the true signal (the gradient you'd get by averaging over the whole dataset) actually is. A large $B_{\text{noise}}$ means individual gradients are noisy relative to the signal, so a large batch is needed to average that noise down enough to see the true direction clearly; a small $B_{\text{noise}}$ means individual gradients already mostly agree, so small batches are already informative and bigger batches add little.

Worked example 1: computing noise scale from two sample gradients

Two training examples produce per-example gradients (for a single scalar parameter) of $g_1 = 3$ and $g_2 = 5$. The mean gradient is $G = 4$, so $|G|^2 = 16$. The variance across examples is $\frac{(3-4)^2+(5-4)^2}{2} = 1$, so $\text{tr}(\Sigma) \approx 1$ in this toy one-parameter case. That gives $B_{\text{noise}} \approx 1/16 = 0.0625$ — a small ratio, meaning individual gradients are already fairly consistent with the true direction here, so small batches would already give a usably clean signal, and there is little to gain from batching them up further.

Worked example 2: doubling batch size below versus above the critical point

Suppose a real training run has an estimated $B_{\text{noise}} = 256$. Doubling batch size from 64 to 128 — both still well below 256 — roughly halves the noise in the gradient estimate and correspondingly allows close to twice as many useful steps to be taken per unit of wall-clock time, since noise was the binding constraint: near-linear speedup. Doubling batch size from 1,024 to 2,048 — both already well above 256 — buys almost no additional convergence speed per step, because the gradient estimate was already precise enough that noise was no longer the bottleneck; the extra compute in the larger batch is mostly wasted relative to simply running more, smaller-batch steps in the same time.

The jitter visible in a noisy descent path is a direct visual of gradient noise — increasing batch size smooths that jitter, but only up to the point where the path was already about as smooth as the underlying loss surface allows.

What this means in practice

This is the practical basis for choosing batch sizes in large-scale training: teams estimate $B_{\text{noise}}$ early in a training run and scale batch size (and data-parallel hardware) up to roughly that point to get the best return on additional compute, since scaling further mostly burns extra hardware for negligible extra convergence speed per unit of data processed.