Batch, Mini-Batch and Online Training

Every gradient descent step needs a gradient, and a gradient is an average of how the loss changes over some set of examples. The question "which examples?" has three standard answers, and they are not just an implementation detail — they change how noisy each step is, how much it costs, and how quickly the model can absorb new data as it arrives.

The analogy: polling the whole country, a focus group, or one voter

Predicting an election by surveying every single voter gives you the exact, noise-free answer — but takes forever and you can't do it every hour. Surveying a random focus group of a few hundred people gives you a decent, slightly noisy estimate, fast enough to repeat many times a day. Asking one random voter on the street gives you an extremely noisy read on any single ask, but you can do it continuously, updating your estimate the instant you get each new answer. These are batch, mini-batch, and online (stochastic) training, respectively — same target, three different sampling strategies with different costs and different noise.

The three regimes

$$\theta \leftarrow \theta - \eta \, \nabla_\theta \frac{1}{|B|}\sum_{i \in B} \mathcal{L}(x_i, \theta)$$

This is the same update rule in all three cases — a step of size $\eta$ against the average gradient over some batch $B$ of examples — and the only thing that changes is $|B|$: batch ($B$ = entire dataset, one gradient computed per full pass) is exact but expensive per step and cannot react to new data until the whole set is re-processed; mini-batch ($B$ = a random subset, typically 32–512 examples) is the default in practice, trading a small amount of gradient noise for many more, cheaper steps per pass; online ($|B| = 1$, updating on each example as it streams in) has the noisiest individual steps but reacts to new information immediately, which matters when the underlying data-generating process is itself shifting over time.

Worked example 1: gradient noise, concretely

Suppose true individual-example gradients for a single weight are $\{4, -2, 6, 0, -8, 3, 5, -1\}$ (true average $= 0.875$). A batch step over all 8 uses the exact average, $0.875$. A mini-batch of 4 randomly drawn examples, say $\{4, -2, 6, 0\}$, averages to $2.0$ — noticeably off from $0.875$ purely from sampling. An online step using just one example, say $-8$, moves the weight in a direction opposite the true average entirely. None of these are "wrong" — each is an unbiased estimate of the true gradient in expectation — but their variance around that true value shrinks as batch size grows, exactly like a poll's margin of error shrinking with sample size.

Worked example 2: steps per pass and total compute

A dataset has 10,000 examples. Batch training does 1 gradient computation per epoch (using all 10,000), so after 50 epochs it has taken 50 gradient steps total, each using all 10,000 examples — 500,000 example-evaluations, 50 updates. Mini-batch training with batch size 100 does $10{,}000/100 = 100$ steps per epoch, so 50 epochs give 5,000 updates using the same 500,000 example-evaluations total — 100 times as many weight updates from the identical amount of data seen. That is the real reason mini-batch dominates practice: far more opportunities to correct course per unit of compute, at the cost of each individual correction being noisier.

Compare how a single noisy step above can overshoot compared to a smoother, well-averaged one — mini-batch training looks like many small, slightly-wobbly versions of that same step, correcting on each pass.

What this means in practice

For a trading model retrained on new market data, online or small mini-batch updates let it adapt quickly to a regime change, at the cost of being more easily whipsawed by noise on any single update — while full-batch retraining is stable but stale between refreshes. In practice, mini-batch is the default for offline model training because GPU hardware processes a moderate batch nearly as fast as a single example, so there is little reason to pay full-batch's per-step cost or accept pure online's per-step noise.