Stochastic Gradient Boosting

Ordinary gradient boosting fits every new tree using every training row, correcting the ensemble's current mistakes as precisely as it can. That precision is also a liability: a tree that sees the exact same rows every round will happily fit noise in those rows just as eagerly as it fits real signal, and the ensemble compounds that noise-fitting round after round. Stochastic gradient boosting fixes this with a small change — feed each new tree only a random subsample of the rows.

The analogy: polling a random sample instead of asking everyone every time

Imagine a pollster who, before every single update to their forecast, re-interviews the entire population — that forecast will track every quirk of every respondent, including ones who answered on a bad day. A more robust pollster instead surveys a fresh random subset each time; over many rounds, idiosyncrasies specific to any one respondent get diluted, because they only show up in some rounds, not all of them, while the genuine trend shows up in every random subset. Stochastic gradient boosting is that second pollster applied to trees: each tree only sees a random slice of the data, so no single tree can lock onto any one row's quirks too hard.

The mechanism

At each boosting round, before fitting the next tree to the current residuals (or gradients), stochastic gradient boosting draws a random subsample of rows — typically without replacement, at a fraction $\eta$ (commonly $0.5$ to $0.8$) of the full training set — and fits that round's tree only on the sampled rows. The next round draws a fresh, independent subsample. Formally, if $g_i$ is the gradient for row $i$ at round $t$, the tree at round $t$ minimizes a loss computed only over $i \in S_t$, a random subset:

$$\text{tree}_t = \arg\min_{f} \sum_{i \in S_t} \left(g_i - f(x_i)\right)^2$$

In plain English: instead of every tree seeing the whole training set's mistakes, each tree sees only a random slice of them, and it is a different random slice each round.

Worked example 1: a noisy row's influence shrinks

Suppose row 47 has a corrupted label that produces an unusually large gradient. With no subsampling, row 47 appears in all $T$ rounds — say $T=200$ — so every one of the 200 trees gets a chance to overfit to it. With subsampling at rate $0.6$, row 47 only appears in roughly $0.6 \times 200 = 120$ rounds, and more importantly, in each of those rounds it is competing to define split points with a different random mix of neighbors, so no fixed pattern of "always compensate for row 47" can build up across the ensemble the way it can without subsampling.

Worked example 2: subsample fraction and effective diversity

Take a training set of 1,000 rows and compare subsample fractions of $1.0$ (no subsampling), $0.5$, and $0.1$. At $1.0$, every tree is built from the identical 1,000 rows — trees only differ because of what earlier trees already corrected for. At $0.5$, each tree sees a different random 500 rows, so two trees built at the same round on two different bootstrapped runs would look meaningfully different from each other — this extra diversity across trees, plus the speed gain from fitting on half the data, is the direct trade quant teams make. At $0.1$, each tree sees only 100 rows — fast and highly diverse, but individual trees can become too noisy to be useful, and validation performance typically starts to degrade past some point, which is why the fraction is tuned, not maximized.

Drag model complexity in the explorer above: subsampling nudges a gradient-boosted ensemble left along that curve — a little less fit to training noise, usually a little better on held-out data — which is exactly the bias-variance trade this technique is buying.

What this means in practice

Subsample fraction is one of the first knobs quants tune in gradient boosting libraries, typically alongside column subsampling (sampling features per tree or per split, the analogous trick applied to columns instead of rows). Both are usually cheap to try and often improve out-of-sample performance meaningfully, especially on datasets with noisy labels — common in financial data, where the "label" itself (a forward return) is inherently noisy.