Gradient Accumulation

Larger batch sizes generally give a smoother, more reliable gradient estimate and let training use hardware more efficiently, but a big batch may simply not fit in GPU memory alongside a large model. Gradient accumulation gets the statistical benefit of a large batch without the memory cost: instead of computing gradients on one huge batch and updating immediately, the model runs several smaller "micro-batches" forward and backward in sequence, adding each micro-batch's gradients into a running total, and only calls the optimizer's update step once the desired number of micro-batches have been accumulated. The parameters see exactly the update they would have seen from one large batch, computed with the memory footprint of the smallest micro-batch, at the cost of doing the same number of forward/backward passes sequentially rather than in parallel.

For example, if a GPU can only fit a batch of 32 samples but the target effective batch size is 256, the training loop runs eight micro-batches of 32, accumulating gradients across all eight before triggering one optimizer step — reaching the same effective batch size as a single 256-sample step, just with eight times the wall-clock time per update rather than eight times the memory. One detail matters for correctness: the accumulated gradients need to be averaged (or the loss scaled down) by the number of micro-batches before the update, otherwise the effective step size ends up eight times too large and training can diverge instead of matching the large-batch behaviour it was meant to reproduce. Most training frameworks handle this scaling automatically once accumulation is turned on, but it is a common source of bugs when implemented by hand. Gradient accumulation is often combined with mixed-precision training and multi-GPU data parallelism, since all three techniques address the same underlying constraint from different angles: fitting a target effective batch size into a fixed amount of available memory and compute.