Tuning Random Forests: Trees, Depth and mtry

A random forest is a bagged ensemble of trees with one extra trick: at every split, instead of considering all features, it considers only a random subset of size $mtry$. That addition, on top of bagging's bootstrap resampling, is what makes random forests behave differently from plain bagged trees, and it comes with its own hyperparameters. Compared to gradient boosting's long list of interacting knobs, a random forest has a short one: number of trees, depth, and $mtry$.

The analogy: a committee forced to look away

Bagging alone is a committee where each member sees a slightly different slice of evidence but is free to weigh every argument. Random forests add a twist: at every decision point, each member is forced to consider only a random handful of arguments, ignoring the rest even if their favorite is excluded that time — forcing different members down different reasoning paths even on similar evidence.

The mechanics: three knobs, three different jobs

Number of trees ($M$): unlike boosting, more trees essentially never overfits — each is grown independently, and averaging reduces variance monotonically, so $M$ is mostly a compute-versus-diminishing-returns call, flattening out somewhere between 100 and 1000. Depth / minimum leaf size: forests traditionally grow trees fully, relying on averaging across decorrelated trees to control variance, though capping depth can help on noisy data. $mtry$: features randomly sampled per split, typically $\sqrt p$ for classification or $p/3$ for regression. Smaller $mtry$ forces more diversity (lower $\rho$) at the cost of weaker individual trees, since good features get excluded more often; larger $mtry$ (up to $mtry=p$, plain bagging) gives stronger individual trees but less diversity.

$$\text{Var}(\bar f) = \rho(mtry)\,\sigma^2 + \frac{1-\rho(mtry)}{M}\sigma^2 .$$

In plain English: $mtry$ directly controls the correlation term $\rho$ — shrinking it pushes $\rho$ down but can raise each tree's own variance if it's forced to ignore genuinely strong features too often; the sweet spot balances the two.

Worked example: sweeping mtry on a 20-feature dataset

Predicting bond downgrade with $p=20$ features: default $mtry=\sqrt{20}\approx4$ gives out-of-bag error 12%. $mtry=20$ (plain bagging) raises it to 15% — every tree keeps choosing the same dominant features at the top split, making trees highly correlated. $mtry=1$ raises it to 17% the other way — trees are so restricted that many build genuinely weak splits, and higher individual error outweighs the diversity gain. The default $\sqrt p$ region sits close to the actual sweet spot here, typical though still worth confirming with a small grid search.

Think of $mtry$ as sliding along the complexity axis above: too low pushes trees toward high individual bias (weak splits forced by restriction), too high pushes the ensemble toward high correlation — either way test error rises, with a minimum in between.

What this means in practice

Start from standard defaults (unpruned trees, a few hundred of them, $mtry=\sqrt p$) and treat $mtry$ as the one hyperparameter most worth tuning via cross-validation or out-of-bag error, since it directly trades tree strength against diversity; tree count mostly just needs to be large enough that error has flattened.