Ensemble Methods

An ensemble is a committee of models whose votes are combined into one prediction. The reason committees work is the same reason a crowd's average guess of the jar's jelly-bean count beats almost every individual: independent errors cancel. If each model is a little wrong in a different direction, averaging them washes the mistakes out and keeps the shared signal. This is why the winning entries in prediction contests are almost always ensembles, and why a random forest, itself an ensemble of trees, is a sensible default model.

The trick has a precise statistical basis in the bias-variance decomposition. A single flexible model tends to have low bias but high variance, it fits the training data well but wobbles a lot from sample to sample. Averaging many such models keeps the low bias while slashing the variance, provided the models are not all making the same mistake.

The variance-correlation identity

Average $T$ models, each with error variance $\sigma^2$ and pairwise correlation $\rho$ between their errors. The variance of the averaged prediction is

$$\operatorname{Var}(\bar f) = \rho\,\sigma^2 + \frac{1-\rho}{T}\,\sigma^2.$$

Read this carefully, it is the whole theory of ensembling in one line. The second term shrinks toward zero as you add models ($T \to \infty$), so more models always help, up to a point. But the first term, $\rho\sigma^2$, does not depend on $T$: it is a floor set by how correlated the models are. If every model makes the same error ($\rho = 1$), averaging does nothing. The entire game is decorrelating the members so that floor is low.

The three families

Method	Attacks	How	Example
Bagging	Variance	Train models on bootstrap resamples, average them	Random forest
Boosting	Bias	Train models sequentially, each fixing the last's errors	XGBoost, AdaBoost
Stacking	Both	Train a meta-model to blend the base models' outputs	Blended competition entries

Bagging (bootstrap aggregating) resamples the data with replacement, fits a model on each resample, and averages. Because each model sees a slightly different dataset, their errors decorrelate, and the average has lower variance. Boosting goes the other way: it builds models one at a time, each new one focusing on the observations the running ensemble still gets wrong, which drives down bias aggressively but risks overfitting on noisy data. Stacking trains a second-level model to learn how to combine the base predictions, rather than just averaging or voting.

Worked example: the wisdom of a crowd

Suppose you have five classifiers, each with 60% accuracy on a binary up/down call, and, crucially, their errors are independent. Take a majority vote. The ensemble is correct whenever at least 3 of the 5 agree with the truth, which follows a binomial:

$$P(\text{correct}) = \sum_{k=3}^{5} \binom{5}{k}(0.6)^k (0.4)^{5-k}.$$

Computing the three terms: $\binom{5}{3}(0.6)^3(0.4)^2 = 10 \times 0.216 \times 0.16 = 0.346$; $\binom{5}{4}(0.6)^4(0.4) = 5 \times 0.1296 \times 0.4 = 0.259$; $\binom{5}{5}(0.6)^5 = 0.078$. Adding: $0.346 + 0.259 + 0.078 = 0.683$. Five independent 60% models vote their way to 68% accuracy, and with more independent models the accuracy keeps climbing toward 1.

The catch is that "independent" is doing enormous work. In markets, models built on the same features and the same history make highly correlated errors, so $\rho$ is large and the curve above flattens fast. The practical route to diversity is to vary the ingredients: different feature sets, different algorithms, different lookback windows, different resamples.

Pitfalls in finance

• Correlated members buy little. Five variations of the same gradient-boosting model share their mistakes, so the ensemble barely beats one of them. Force diversity or expect disappointment.
• Boosting memorizes noise. On low-signal, non-stationary market data, boosting can drive training error to zero by fitting quirks that never recur, see Tree Ensembles in Finance. Regularize hard: shallow trees, strong shrinkage, early stopping.
• Stacking leaks. If the meta-model is trained on the same data the base models saw, it learns their in-sample fits, a classic leak. Train the blender on out-of-fold predictions only.
• False confidence. A tighter ensemble prediction looks more certain, but if all members share a blind spot (a regime they never saw), the ensemble is confidently wrong. Low variance is not low bias.

Ensembles are the quiet workhorses of applied machine learning: gradient boosting and random forests are both ensembles, and even a hand-blended average of a linear model and a tree often beats either alone. The one principle to carry: build members that are individually decent and collectively diverse, because the ensemble's power comes entirely from the mistakes its members do not share.