Gradient Boosting as Functional Gradient Descent

A golfer does not try to hole every shot from the tee. They hit the ball as well as they can, look at where it landed relative to the hole, and hit again — this time aiming to correct exactly the distance and direction still remaining. Each shot is a small, targeted fix for whatever error the previous shots left behind, and a handful of imperfect shots, each correcting the last, gets the ball in the hole more reliably than trying to nail an impossible single stroke. Gradient boosting builds a model the same way: instead of fitting one model that tries to get everything right at once, it fits a short sequence of weak models, each one aimed squarely at whatever the current combination still gets wrong.

From weight updates to function updates

Ordinary gradient descent updates a vector of numbers — the weights $\theta$ of a model — by stepping a little in the direction that most reduces a loss: $\theta \leftarrow \theta - \eta \nabla_\theta L$. That direction is a vector, because $\theta$ is a vector.

Gradient boosting asks the same question about the model's predictions themselves, treated as one giant object to be optimized directly, without ever parameterizing them by a fixed set of weights. At round $m$, the current model is a function $F_{m-1}(x)$ — everything built so far, added together. The next round asks: which small function, added on top of $F_{m-1}$, would most reduce the loss? That small function is found by computing the negative gradient of the loss with respect to the current predictions, one number per training point, and then training a weak learner — almost always a shallow decision tree — to approximate that vector of numbers as a function of $x$. The tree is then added in:

$$F_m(x) = F_{m-1}(x) + \eta \, h_m(x)$$

In words: the new model equals the old model, plus a small step (controlled by a learning rate $\eta$) in the direction of a freshly fit tree $h_m$, where that tree was trained specifically to point toward "what would most reduce the loss from here." This is gradient descent — same update rule, same "small step in the downhill direction" logic — except each step is an entire tree rather than a nudge to a fixed set of numbers, which is why it is called functional gradient descent.

Squared error makes the gradient a familiar object

For squared-error loss $L = \frac{1}{2}(y - F(x))^2$, the negative gradient with respect to $F(x)$ works out to

$$-\frac{\partial L}{\partial F(x)} = y - F(x)$$

In words: for the most common loss function, the "negative gradient" that boosting fits at each round is nothing more exotic than the residual — actual value minus current prediction. This is why the plainest description of gradient boosting is "repeatedly fit a small tree to whatever error is still left over," and it is also literally, exactly, gradient descent in disguise.

Worked example 1: two boosting rounds by hand

Four points: $x = 1,2,3,4$ with targets $y = 2,4,5,9$. Boosting starts from the simplest possible model, the mean: $F_0(x) = \bar y = \frac{2+4+5+9}{4} = 5$ for every $x$.

Round 1. Residuals, $y - F_0$: $-3, -1, 0, 4$. A shallow tree finds the split $x \le 2$ vs. $x > 2$ that best separates these: the left group averages $\frac{-3-1}{2} = -2$, the right group averages $\frac{0+4}{2} = 2$. With learning rate $\eta = 1$:

$$F_1(x) = \begin{cases} 5 + (-2) = 3 & x \le 2 \\ 5 + 2 = 7 & x > 2 \end{cases}$$

Sum of squared errors falls from $F_0$'s $9+1+0+16=26$ to $F_1$'s $(3{-}2)^2+(3{-}4)^2+(7{-}5)^2+(7{-}9)^2 = 1+1+4+4 = 10$.

Round 2. New residuals, $y - F_1$: $-1, 1, -2, 2$. This time the best split is $x \le 3$ vs. $x > 3$: the left group ($x=1,2,3$) averages $\frac{-1+1-2}{3} = -0.667$, the right group ($x=4$) is just $2$.

$$F_2(x) = \begin{cases} 3 - 0.667 = 2.333 & x = 1,2 \\ 7 - 0.667 = 6.333 & x = 3 \\ 7 + 2 = 9 & x = 4 \end{cases}$$

Sum of squared errors: $(2.333{-}2)^2 + (2.333{-}4)^2 + (6.333{-}5)^2 + (9{-}9)^2 = 0.111 + 2.778 + 1.778 + 0 = 4.667$.

$$\text{SSE: } 26 \;\to\; 10 \;\to\; 4.667$$

Each round fit a small tree to the residual left over from the previous round, exactly the negative-gradient object the formula above says it should, and each round's addition drove the total squared error down.

Worked example 2: the gradient looks different for a different loss

For classification with log loss, the gradient is not the raw residual but $y - p$, where $p$ is the current model's predicted probability (after passing $F(x)$ through a sigmoid). Three points with true labels $y = 1, 0, 1$ and current predicted probabilities $p = 0.6, 0.4, 0.7$:

$$\text{pseudo-residuals} = y - p = (1-0.6,\; 0-0.4,\; 1-0.7) = (0.4,\; -0.4,\; 0.3)$$

A tree is fit to these three numbers exactly as before — the mechanism is identical, only the formula defining "what counts as the gradient" changed, because it changed with the loss function. This is the general recipe: pick any differentiable loss, compute its negative gradient with respect to the current predictions, and boosting fits trees to chase that gradient down to zero, round after round.

The explorer above shows ordinary gradient descent taking steps down a loss curve — small steps that converge, or overshoot if the step size is too large. Gradient boosting is the same picture with one difference: instead of a step being "move this one number a little," a step is "add this whole tree." Drag the step size in the explorer and notice the same overshoot risk that a poorly tuned learning rate creates in boosting — too large a learning rate and each added tree overcorrects the previous round's error rather than gently reducing it.

What this means in practice

This view explains two of gradient boosting's most important knobs directly. The learning rate $\eta$ is exactly the step size in a gradient descent update — too large and successive trees overcorrect and oscillate, exactly as an overshooting step does in ordinary gradient descent; too small and training needs far more rounds to converge. The number of boosting rounds is exactly the number of gradient steps taken, and taking too many is the direct boosting analogue of training a model for too many epochs — it starts fitting the noise in the residuals rather than genuine signal, which is why boosted models are almost always paired with early stopping on a held-out validation set. Because each round only ever sees the residual, not the raw target, a single mislabeled or outlier data point can dominate several rounds' worth of pseudo-residuals — robust loss functions (Huber loss instead of squared error) exist specifically to blunt that.

Practice

1. Continuing worked example 1 for a third round, the residuals after $F_2$ are $-0.333, 1.667, -1.333, 0$. What would you expect the sum of squared error to do — keep falling, flatten, or start rising — if boosting continued for many more rounds without a validation check?
2. Why does gradient boosting with squared-error loss fit each tree to the literal residual $y - F(x)$, while gradient boosting with log loss fits each tree to $y - p$ instead?
3. What single hyperparameter plays the same role in boosting that the step size $\eta$ plays in ordinary gradient descent, and what goes wrong if it is set too high?