Leaf-Wise vs Level-Wise Tree Growth
Level-wise tree growth expands every leaf at the current depth before going deeper, while leaf-wise growth always splits whichever single leaf would reduce error the most, producing deeper, more irregular but often more accurate trees faster.
Prerequisites: Oblique Trees and Linear Model Trees
Most classic decision tree algorithms, and gradient boosting libraries like XGBoost by default, grow trees level-wise: every leaf at the current depth gets split before any leaf one level deeper is considered, producing a balanced, symmetric tree. LightGBM popularized an alternative: leaf-wise growth, which always splits whichever single leaf across the whole tree offers the biggest reduction in loss, regardless of depth.
Leaf-wise growth reaches a given loss reduction with fewer splits than level-wise growth, because it never wastes a split on a leaf that wouldn't help much — but it can grow one branch much deeper than others, so it needs an explicit max-depth or max-leaves limit to avoid overfitting on small datasets.
Why it matters in practice
A level-wise tree treats every leaf at a given depth as equally worth splitting, even if some leaves are already nearly pure and gain little from a further split. A leaf-wise tree instead always chases the single most promising leaf, which tends to converge to lower training error faster for the same number of leaves — this is a large part of why LightGBM is often faster than XGBoost's default setting at a comparable accuracy. The tradeoff is that leaf-wise trees are more prone to overfitting on small datasets, since nothing stops the algorithm from repeatedly deepening one branch that happens to fit noise well, which is why LightGBM exposes a max_depth or num_leaves cap as an important tuning knob.
Worked example
Given a budget of 8 leaves, a level-wise tree grows a full, balanced tree 3 levels deep (splitting every leaf at each level: 1 → 2 → 4 → 8). A leaf-wise tree with the same 8-leaf budget might instead split one branch down to depth 5 while leaving a nearby, already-pure branch unsplit at depth 1 — reaching lower training loss with the same leaf count, but risking overfitting if that deep branch was chasing noise in a small training sample.
Further reading
- Ke et al. (2017), 'LightGBM: A Highly Efficient Gradient Boosting Decision Tree'