Blending vs Stacking

You have trained three separate models on the same problem — say, a gradient-boosted tree, a linear model and a small neural net predicting next-week stock returns. Each has different strengths and different mistakes. Averaging their outputs usually beats any one of them, but a plain average treats all three as equally trustworthy everywhere, which is rarely true. What you actually want is a rule that says "trust the tree more when volatility is low, trust the linear model more when it's high." Learning that rule from data, instead of guessing it, is what stacking and blending do.

Both approaches train a meta-model whose inputs are the predictions of the base models, not the original features. The meta-model learns how to weight and combine those predictions. The entire difference between the two names is which data the meta-model is allowed to see.

Blending: one holdout split

Blending splits the training set once into two pieces. The base models are fit on the first piece and used to predict on the second, held-out piece. Those held-out predictions, together with the true held-out labels, train the meta-model. Because the meta-model never saw predictions the base models made on data they were trained on, there is no leakage — but you have thrown away a chunk of data twice: the base models never see the holdout, and the meta-model only ever learns from that one holdout slice.

Stacking: out-of-fold predictions

Stacking does the same job with cross-validation instead of a single split. Split the training data into $K$ folds. For each fold, train the base models on the other $K-1$ folds and predict on the held-out fold. After cycling through all folds, every training row has an out-of-fold prediction from each base model — a prediction made by a version of the model that never saw that row. Stack these out-of-fold predictions as features and train the meta-model on the full dataset. Finally, retrain each base model on all the training data for use at prediction time.

This uses every row for both stages, which blending does not, at the cost of training each base model $K$ times instead of once.

Stacking (K = 4) each fold held out once → every row gets an out-of-fold prediction, all rows used

Worked example

Suppose you have 10,000 daily observations and three base models. With blending you might use 7,000 rows to train the base models and the remaining 3,000 to train the meta-model — the meta-model only ever learns from those 3,000 rows. With 5-fold stacking, all 10,000 rows get an out-of-fold prediction from each base model (each row's prediction comes from a model trained on the other 8,000), so the meta-model trains on 10,000 rows. The base models are each trained 5 times during the out-of-fold pass, then once more on the full 10,000 for deployment — six training runs per base model instead of one.

Why leakage is the whole risk

The reason not to just fit base models and a meta-model on the same data is that the meta-model would learn to trust whichever base model overfit hardest, since an overfit model's training predictions look artificially accurate. Both blending and stacking exist to break that link by making sure the meta-model only ever sees predictions made on data the base model did not train on. Get this wrong — feed the meta-model in-sample predictions — and your backtest will look excellent and your live results will not, which is the same failure mode as Backtest Overfitting.

In practice, quants lean toward blending when data is abundant and retraining is expensive, and toward stacking when data is scarce enough that discarding a holdout chunk meaningfully hurts the base models. Both beat a naive average whenever the base models make genuinely different kinds of mistakes, which is the same condition that makes any ensemble worth building in the first place — see Why Ensembles Work: The Ambiguity Decomposition.

Choosing the meta-model

The meta-model itself is usually kept simple on purpose — a linear or logistic regression over the base models' predictions is the common default, occasionally a shallow tree if the interactions between base models are expected to be non-linear. This is deliberate: the meta-model only sees a handful of columns (one per base model), so it has little room to overfit even with a flexible learner, but a heavily tuned meta-model can still learn to lean on whichever base model happened to look best on the particular holdout or out-of-fold slice it was trained on, rather than on a genuinely more reliable model. Keeping the meta-model simple, and checking that its learned weights are stable across different random seeds for the fold split, is a cheap guard against that.

A second practical decision is what exactly to feed the meta-model besides raw predictions. Many stacking pipelines add the original features back in alongside the base models' outputs, letting the meta-model learn conditional trust — "lean on the tree model when volatility is high, the linear model otherwise" — rather than a single fixed blend. This closes some of the gap with fully joint modelling approaches, at the cost of reintroducing the original feature set's own overfitting risk into the final stage, so it deserves the same out-of-fold discipline as everything else in the pipeline.