DART: Dropout for Boosted Trees

Ordinary gradient boosting builds trees one at a time, and each new tree is trained to fix the errors left by all the trees before it — which means the very first few trees, built when the errors were largest, end up carrying outsized influence over every later prediction. DART borrows the dropout trick from neural networks to fix this: on each boosting round, it randomly hides a subset of the already-built trees before computing what the new tree should correct, then rescales things so removing those trees didn't distort the running total.

Worked example. After building 50 trees in a standard gradient boosting model, imagine tree #2 happens to capture a strong, simple pattern and ends up contributing a disproportionate share of every prediction from then on; trees built after it mostly make tiny corrections around that pattern. Under DART with a 10% dropout rate, roughly 5 of those 50 trees are randomly muted before each new tree is fit — so on some rounds tree #2 itself gets excluded, forcing the new tree to learn to stand in for it rather than nudge around it. Over many rounds this spreads predictive weight more evenly across the ensemble.

The cost is that DART trees can no longer be evaluated incrementally as cheaply as standard boosting, since which trees are "active" changes randomly each round, making training somewhat slower — a worthwhile trade when a model is overfitting and shrinkage or early stopping alone aren't enough.