Out-of-Fold Predictions for Stacking

Stacking combines several different models — say, a random forest, a boosted tree ensemble, and a linear model — by training a final "meta-model" that learns how to weigh and combine their predictions into one better prediction. The idea sounds simple: run the base models, collect their predictions, use those as features for the meta-model. Done naively, this fails badly, because a base model's prediction on a training example it was trained on is not a fair sample of how well it generalizes — and the meta-model will happily learn to over-trust whichever base model overfit its training data the hardest.

Why naive stacking leaks information

If a base model is trained on the full training set and then asked to predict on that same set, its predictions will be unrealistically accurate — memorization, not genuine skill. Using those training-set predictions as input features for the meta-model means it sees a version of each base model that looks far better than it will actually perform on new data, and learns to lean heavily on whichever base model overfit the most. The meta-model has effectively learned to trust overfitting, and the stacked ensemble can perform worse than a single well-tuned base model.

The fix: out-of-fold predictions

The standard fix borrows the machinery of k-fold cross-validation. Split the training data into $K$ folds. For each fold $k$, train each base model on the other $K-1$ folds and predict on the held-out fold $k$ — data that model never saw during that particular training run. Doing this for every fold produces, for every single training example, a prediction from a base model that was never trained on that example. These are the out-of-fold (OOF) predictions, and they're what gets fed into the meta-model as training features — an honest, unbiased estimate of how each base model actually performs on unseen data, for every row in the training set simultaneously.

In plain English: instead of asking each base model to grade its own homework, you rotate through the data so every prediction used to train the meta-model came from a base model being tested, not from a base model recalling an answer it already memorized.

Worked example: 5-fold OOF construction

Suppose you have 1,000 training rows split into 5 folds of 200 each, and two base models (a random forest and a gradient-boosted tree). For fold 1 (rows 1–200): train both base models on rows 201–1000, then predict on rows 1–200. Repeat through fold 5. After all five rounds, every row has exactly one out-of-fold prediction from each base model — none produced by a model that had seen that row during training. Stack these 1,000 rows of (OOF prediction 1, OOF prediction 2, true label) and fit a simple meta-model on top: it learns, honestly, how much weight to place on each base model. Once fit, the base models are retrained one final time on the full training set for use on genuinely new data.

Rotating which fold is held out is conceptually similar to how the explorer above resamples different random paths from the same process — each fold gives you an independent, honest look at what the base model does on data it hasn't fit, rather than one path (the training set) that's been directly memorized.

What this means in practice

Any time you build a stacked ensemble — combining a boosted tree model with a neural net and a linear model for a signal-combination layer — insist on out-of-fold predictions for the meta-model's training data, never predictions from base models evaluated on their own training rows. This is a common bug in quickly-assembled ensemble pipelines, easy to miss because the overfit numbers look great in-sample and only surface as underperformance later, out-of-sample.