Early Stopping in Boosting

Every round of gradient boosting adds one more tree, and each new tree always makes the model fit the training data at least a little better. Left unchecked, this eventually starts fitting noise rather than signal. Early stopping watches the model's error on a separate validation set after every round and halts training the moment that validation error stops improving, even though training error would keep falling if allowed to continue.

Worked example. A gradient boosting classifier is trained for up to 1,000 rounds, checking validation log-loss every 10 rounds. Validation loss drops steadily through round 200, plateaus between rounds 200 and 260, and starts creeping upward after round 260 as the trees begin fitting training-set idiosyncrasies. With a "patience" setting of 50 rounds (stop if no improvement for 50 rounds), training halts around round 310 and the model is rolled back to its round-260 state, the point of best validation performance, rather than continuing all the way to round 1,000.

Early stopping is popular because it is essentially free regularization: instead of separately tuning the number of trees as a hyperparameter through repeated cross-validation, a single training run with a validation set finds a good stopping point automatically, and it works well in combination with shrinkage and row/column subsampling.