A Practical Tuning Order for Gradient Boosting

A gradient boosting library exposes ten or more hyperparameters, and a naive grid search over all of them simultaneously is combinatorially enormous — five values each for eight parameters is nearly 400,000 combinations. Worse, tuning them all at once makes it hard to tell which knob actually mattered, because results are entangled. A practical tuning order fixes most parameters, sweeps one or two at a time, and moves on, roughly in order of how much each one tends to matter.

The analogy: tuning a car, not rebuilding the engine blind

A mechanic tuning a car's performance does not randomly adjust every dial simultaneously and hope for the best — they check tire pressure first (cheap, high-impact), then alignment, then engine timing, each time holding everything else fixed so they can tell what actually changed the result. A haphazard, all-at-once approach to gradient boosting hyperparameters has the same problem a haphazard mechanic has: you end up unable to say which change helped, and you have wasted far more time than a staged approach would have taken.

The order, and why it is ordered this way

Step 1 — learning rate and number of trees together. Fix a small learning rate (commonly $\eta \approx 0.05$–$0.1$) and let early stopping on a validation set choose the number of trees. This pair is tuned jointly, not independently, because they trade off directly against each other (see Stochastic Gradient Boosting and Regularization Knobs in Gradient Boosted Trees), and getting it roughly right first stabilizes every subsequent step.

Step 2 — tree structure: max depth and min child weight. These control how elaborate any single tree can be, and they interact strongly with each other but weakly with most other parameters, which is why they are tuned as a pair before anything else structural.

Step 3 — sampling: row and column subsample fractions. Once tree shape is roughly right, tune how much data and how many features each tree sees. This step usually delivers a smaller improvement than steps 1–2, but a real one, especially against overfitting.

Step 4 — leaf-level regularization: $\gamma$, $\lambda$, $\alpha$. These fine-tune how eagerly the tree grows and how extreme leaf values are allowed to be — typically the smallest marginal gains, tuned last because their effect is dwarfed by getting steps 1–3 wrong.

Worked example 1: what happens tuning in the wrong order

Suppose a quant fixes $\eta = 0.3$ (high) from the library default and immediately grid-searches max depth from 3 to 10, without revisiting $\eta$. At $\eta=0.3$, deep trees overfit almost immediately regardless of depth, so the search reports "depth 3 is best" — but that conclusion is an artifact of the too-high learning rate, not a real property of the data. Redo the search with $\eta=0.05$ and early stopping, and the "best" depth might shift to 5 or 6, because the ensemble no longer overfits as fast at any given depth. Tuning depth before fixing $\eta$ produced a misleading answer.

Worked example 2: a concrete staged search

Start: $\eta=0.05$, everything else at library defaults; early stopping on a validation fold picks 850 trees. Step 2: sweep max depth in {3,4,5,6,7} and min child weight in {1,5,10,20} — a 20-point grid, not a 400,000-point one — suppose depth 5, min child weight 10 wins. Step 3: sweep row subsample in {0.6, 0.8, 1.0} and column subsample in {0.6, 0.8, 1.0}, a 9-point grid — suppose 0.8/0.8 wins, improving validation score by a small but real 0.4%. Step 4: sweep $\gamma$ and $\lambda$ over a handful of values — typically the smallest gain of the four steps, often under 0.1%. Total: roughly 30–40 configurations tried, in a staged, interpretable order, versus hundreds of thousands for a blind joint grid.

Each staged sweep above is really just probing a different slice of the complexity axis in the explorer — tree structure moves it fastest, leaf-level penalties move it in small increments at the end.

What this means in practice

This staged order turns an intractable joint search into a sequence of small, cheap grids, and — just as importantly — produces a tuning log a colleague can actually read and sanity-check, rather than an opaque black-box search result. It is not the mathematically optimal search strategy (a proper Bayesian hyperparameter optimizer can do better with enough budget), but it is a reliable, low-compute default.