Data-Parallel Training

Training a large model on one machine means waiting for that one machine to grind through every mini-batch, one after another, for as long as training takes — hours, days, sometimes weeks. The most direct way to speed this up doesn't touch the model or the maths at all: buy more machines and split the data across them. Each machine independently computes gradients on its own slice of the batch, and the only new problem is how to combine what they each found into a single, consistent update.

The analogy: eight analysts grading the same exam, faster together

Imagine grading a thousand exam papers to compute a class average. One grader working alone takes a full day. Give the same thousand papers to eight graders, a hundred and twenty-five papers each, and each one grades their pile in about an hour. To get the true class average, you don't average the eight graders' individual sub-averages naively if the piles differ in size — you sum every grader's point total and divide by the total number of papers. Data-parallel training is exactly this: split the batch across workers, let each compute its own gradient on its slice, then combine those gradients into the single average gradient the model would have computed had one worker processed the whole batch at once.

The mechanics: forward, backward, all-reduce

Each of $K$ workers holds an identical copy of the model's weights $\theta$. A global batch of size $B$ is split into $K$ local mini-batches of size $B/K$. Each worker computes its own local gradient independently:

$$g_k = \frac{1}{B/K}\sum_{i \in \mathcal{B}_k} \nabla_\theta \ell_i(\theta), \qquad g = \frac{1}{K}\sum_{k=1}^{K} g_k$$

In words: each worker averages the gradient over just its own slice of the batch; then all $K$ workers' local averages are themselves averaged together, which is mathematically identical to having computed the gradient over the full combined batch on a single machine. That combining step is called an all-reduce: every worker sends its local gradient to every other worker (efficiently, via a ring or tree communication pattern rather than literally $K^2$ messages) and each ends up holding the same final averaged gradient. Because every worker then applies the identical update to its identical copy of the weights, all $K$ copies stay in sync step after step without ever needing a central coordinator to hold the "real" weights.

Worked example 1: combining four workers' gradients by hand

Four workers each compute a local gradient for a single scalar weight, from their own slice of the batch: $g_1 = 0.42$, $g_2 = 0.38$, $g_3 = 0.55$, $g_4 = 0.29$.

All-reduce averages them: $g = (0.42 + 0.38 + 0.55 + 0.29)/4 = 1.64/4 = 0.41$. Every worker now applies the identical update using $g = 0.41$, e.g. with learning rate $\eta = 0.1$: $\theta_{\text{new}} = \theta - 0.1 \times 0.41 = \theta - 0.041$. Check consistency: this is exactly what a single machine processing the full four-worker-sized batch in one pass would have computed, provided the four local mini-batches were disjoint, equally-sized samples from the same underlying batch — which is the whole point of the scheme.

Worked example 2: throughput and the diminishing-returns wall

A single GPU processes 200 training examples per second. Scale to 8 GPUs with perfect (linear) scaling: $200 \times 8 = 1{,}600$ examples per second, an 8x speedup. In practice, some time is spent on the all-reduce communication step rather than computation. Suppose each step takes $50$ ms of compute and a fixed $10$ ms of unavoidable communication overhead regardless of GPU count, once GPUs must talk to each other at all. Total step time is $60$ ms with communication, effective throughput per GPU: $\frac{1}{0.060} \times \frac{B_k}{1}$ examples per second per worker instead of the ideal $\frac{1}{0.050}$ — a slowdown to $50/60 \approx 83\%$ of ideal per-GPU efficiency, so 8 GPUs deliver roughly $1{,}600 \times 0.83 \approx 1{,}330$ examples per second rather than the full $1{,}600$. Scale to 64 GPUs and the same fixed overhead (now often larger, since more workers means more coordination) eats a proportionally bigger share, which is why data-parallel scaling reliably has diminishing returns well before the naive "just add more GPUs" arithmetic would suggest.

What this means in practice

Data parallelism is the default first move for scaling training, precisely because it changes nothing about the model or the optimisation maths — the only new machinery is the communication step. It has a natural ceiling: it requires every worker to hold a full copy of the model's weights, gradients, and optimiser state in memory, which breaks down once a model is too large to fit on one device at all — that's the point where tensor and pipeline parallelism take over, splitting the model itself rather than the data. It also interacts with the effective batch size: scaling to more workers with a fixed per-worker batch grows the global batch size proportionally, which reduces the gradient noise discussed in SGD Noise and Implicit Regularization and can quietly hurt generalisation unless the learning rate and warm-up schedule are adjusted to compensate — a large-scale training run that looks like a pure infrastructure win can carry a hidden optimisation cost.

In practice, most teams measure a "scaling efficiency" curve before committing to a large worker count: run the same training job at 1, 2, 4, 8, and 16 workers, plot achieved throughput against the ideal linear line, and find the point where the two visibly diverge. Beyond that point, additional hardware is better spent on a second, parallel experiment than on shrinking the same run's wall-clock time further, since the marginal speedup per added GPU has already fallen well below one.