Quant Memo
Core

Random Forests and Out-of-Bag Error

Because each tree in a random forest is trained on a bootstrap resample, roughly a third of the data is left out of any given tree — and scoring each point only on the trees that never saw it gives a free, honest estimate of test error.

random-forestsbaggingmodel-validationensemble-methodsbootstrap

Prerequisites: Bagging and Variance Reduction, Decision Trees and CART

A random forest fits hundreds of decision trees, each on a fresh bootstrap sample — a random draw of the training rows, with replacement, the same size as the original data. Drawing with replacement means some rows get picked two or three times and others aren't picked at all. That leftover portion is not a nuisance; it's a built-in validation set that costs nothing extra to compute.

Every tree in the forest ignores roughly a third of the training rows, purely because of how bootstrap sampling works. Scoring each row only on the trees that never trained on it — its "out-of-bag" trees — produces an error estimate that behaves like held-out test error, without holding anything out.

Why about a third gets left out

For a training set of nn rows, the probability a specific row is not selected in a single bootstrap draw of nn draws-with-replacement is (1−1n)n\left(1 - \frac{1}{n}\right)^n, which converges to e−1≈0.368e^{-1} \approx 0.368 as nn grows. In words: sampling with replacement means a given row misses every one of nn draws about 37% of the time, so on average each tree is trained on about 63% of the unique rows and never sees the remaining 37%.

3 trees, each trained on a bootstrap sample of the same 10 rows Tree 1 Tree 2 Tree 3 filled = in this tree's bootstrap sample · blank = out-of-bag for this tree row 2's OOB prediction uses only Tree 1's vote, since it's the only tree that never trained on row 2
Each tree leaves out a different ~37% of rows; a row's OOB error only pools the trees that never saw it.

Worked example

A forest of 500 trees is grown on 1,000 labeled trades (profitable / not profitable). Row 417 is out-of-bag for about 37% of trees, roughly 185 of them. To get row 417's out-of-bag prediction, only those 185 trees vote: say 140 say "profitable" and 45 say "not." The majority vote, 140-to-45, becomes row 417's OOB prediction, compared against its true label. Repeating this for all 1,000 rows and averaging the mismatches gives the forest's OOB error rate — say 9.4% — without ever touching a separate validation split.

What this means in practice

OOB error is a nearly free stand-in for k-fold cross-validation on i.i.d. tabular data: no extra folds to fit, no data held back from training the final model, and it's available the moment the forest finishes growing. That makes it a convenient dial for tuning hyperparameters like the number of trees or the minimum leaf size — you can watch the OOB error curve flatten as more trees are added and stop growing the forest once it does, without ever touching a held-out test set. It is not a substitute for time-series-aware validation, though — for a forest trained on cross-sectional stock features, a bootstrap sample can still include rows from both before and after a given date, so the "hold out" isn't hold out in the way that matters when the real risk is look-ahead through time.

OOB error assumes rows are exchangeable — that leaving out a random 37% of rows tells you something about generalizing to unseen rows. In financial time series with autocorrelated features (today's momentum score is highly informative about tomorrow's), bootstrap resampling does not break the temporal leakage a true walk-forward split would, so OOB error can look better than live performance actually turns out to be.

Related concepts

Practice in interviews

Statistics →Coding →

Further reading

  • Breiman, 'Random Forests' (Machine Learning, 2001)
  • Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (ch. 15)
ShareTwitterLinkedIn