Random Forests vs Gradient Boosting

Both random forests and gradient boosting are "just a bunch of decision trees" underneath, and beginners often treat them as interchangeable. They are not: the two families attack the exact same building block — a decision tree, which alone tends to overfit badly — from opposite directions, and knowing which direction a given method takes tells you how it fails and how to tune it.

The analogy: independent juries versus a relay of editors

A random forest is like polling many independent juries, each shown a slightly different, randomly resampled version of the same case, and averaging their verdicts — no jury knows what any other jury decided, and the averaging cancels out each jury's individual quirks. Gradient boosting is like a relay of editors reviewing the same document one after another — the first editor does a rough pass, the second editor focuses specifically on what the first one got wrong, the third focuses on what's still wrong after the second, and so on. One approach cancels noise through independent averaging; the other progressively chases down remaining error through correction.

The mechanics, side by side

Random forest: each tree is grown on an independent bootstrap resample of the training data, using a random subset of features considered at each split, and — crucially — every tree is grown fully independently of every other tree; they could all be trained in parallel on different machines. The final prediction is the plain average (or majority vote) across trees:

$$\hat{y}_{RF}(x) = \frac{1}{T}\sum_{t=1}^{T} \hat{y}_t(x)$$

In plain English: grow many overfit-prone trees on different random slices of the data, and let averaging cancel out each individual tree's overfitting, since their errors are only weakly correlated with each other.

Gradient boosting: each tree is grown sequentially, specifically to predict the current residual (or negative gradient) of the ensemble built so far — tree 2 cannot be trained until tree 1 exists, because tree 2's target depends on tree 1's output. The final prediction is a weighted sum, not a simple average:

$$\hat{y}_{GB}(x) = \sum_{t=1}^{T} \eta \cdot \hat{y}_t(x)$$

In plain English: start with a weak model, then add small, targeted corrections one at a time, each one specifically aimed at whatever the ensemble still gets wrong.

Worked example 1: how each handles a noisy label

Suppose one training row has a corrupted label — a return recorded as +8% when it should have been -0.2%. In a random forest, that row appears in roughly 63% of bootstrap samples (the standard bootstrap inclusion rate), so about a third of trees never see it at all, and among the trees that do, it is just one noisy vote diluted by everything else in that tree's random slice — its damage stays fairly contained. In gradient boosting, that row produces an unusually large residual after the first few trees, and every subsequent tree is explicitly built to chase down large residuals — the corrupted row can pull disproportionate attention from many trees in a row, which is exactly why gradient boosting is more sensitive to label noise and typically needs stronger regularization (shrinkage, subsampling, early stopping) to stay robust.

Worked example 2: bias and variance in opposite directions

A single, deep, unpruned decision tree typically has low bias (it can fit almost any training pattern) and high variance (it changes a lot if the training data changes slightly). A random forest keeps roughly the same low bias as a single deep tree, but sharply reduces variance through averaging over decorrelated trees — that is its whole mechanism. Gradient boosting starts from the opposite end: it typically uses shallow, high-bias trees (depth 3–6), and reduces bias progressively as more trees are added, at some risk of variance creeping back up if too many trees or too large a learning rate is used. Same building block, opposite starting point, opposite failure mode.

Drag complexity in the explorer above: a random forest mainly moves down the variance axis as trees are added, while gradient boosting mainly moves down the bias axis as trees are added — the same picture, entered from opposite sides.

What this means in practice

Random forests are a strong, low-fuss default — fast to parallelize, hard to badly overfit through the number of trees alone, and forgiving of default hyperparameters. Gradient boosting usually reaches higher accuracy with careful tuning, but is more sensitive to label noise, requires sequential training, and needs real regularization discipline (see Regularization Knobs in Gradient Boosted Trees) to avoid overfitting.