Gradient Accumulation and Effective Batch Size

A training recipe might specify a batch size of 512 for good reasons — it worked in the original paper, it gives a favorable ratio of gradient noise to signal, it matches the learning rate schedule the recipe was tuned for. But a single GPU's memory might only hold 64 examples at once for that model. Simply switching to batch size 64 changes the actual optimization dynamics, not just a technical detail — smaller batches produce noisier gradient estimates and often need a different learning rate entirely. Gradient accumulation reconciles the two: it processes the 512 examples as eight passes of 64, but only updates the weights once, after summing all eight gradients — numerically equivalent to having run the full batch of 512 at once.

The analogy: paying a bill in installments that still add up to the full amount

A large purchase split across eight monthly installments is still, in total, exactly the same purchase — nothing about the final amount owed changes, only the number of separate transactions used to get there. Gradient accumulation does the same thing to a gradient computation: eight "installments" of 64 examples each, summed together, produce exactly the gradient a single pass of 512 examples would have produced — the model only "pays" (updates its weights) once, after all eight installments are in.

The mechanics

For $k$ accumulation steps, each processing a micro-batch of size $b$, the accumulated gradient before the optimizer step is:

$$g_{\text{total}} = \sum_{i=1}^{k} g_i, \qquad \text{effective batch size} = k \times b$$

In words: run the forward and backward pass on each micro-batch as usual, computing its gradient $g_i$, but instead of immediately calling the optimizer, add $g_i$ into a running total. Only after all $k$ micro-batches have contributed does the optimizer take a single step using the summed gradient — which (for a loss that averages over the batch, as is standard) is mathematically the same update a real batch of size $k \times b$ would have produced, once the accumulated gradient is divided by $k$ to restore the average.

Worked example 1: matching a target effective batch size

Target effective batch size 512, GPU fits 64 examples at a time: $k = 512/64 = 8$ accumulation steps. Each of the 8 micro-batches computes its own gradient via a normal forward and backward pass; none of them individually triggers an optimizer step. After the 8th, the accumulated (and averaged) gradient is used for one optimizer.step() call, and the accumulator is reset to zero for the next group of 8.

Worked example 2: why forgetting to zero the accumulator is a classic bug

Suppose gradients accumulate correctly for $k=4$ steps giving $g_{\text{total}} = 4.0$, and the optimizer step uses $g_{\text{total}}/4 = 1.0$ as intended — correct. But if optimizer.zero_grad() is called only once at the start of training instead of after every step, the next group of 4 micro-batches adds its gradients on top of the stale $4.0$ still in the buffer, say another $3.6$, giving $g_{\text{total}} = 7.6$: an effective gradient of $1.9$ instead of the intended $0.9$ — more than double the correct update size, entirely from a bookkeeping error. Every implementation must zero the gradient buffer immediately after each optimizer step, not just once at the start.

What this means in practice

Gradient accumulation is standard whenever a model or sequence length is too large to fit a target batch size on available hardware — common when fine-tuning large language models on a single GPU. It is not free: it increases wall-clock time roughly proportional to $k$, since the $k$ passes are still done sequentially even though the optimizer step happens once; it also interacts with batch normalization, whose running statistics are computed per micro-batch, so very small micro-batches can leave batch norm statistics noisier than the effective batch size alone suggests.