Stacked Generalization

The simplest way to combine several models' predictions is to average them, or take a fixed weighted average tuned once on a validation set. Stacking goes a step further: instead of a fixed rule, it trains a separate model — the meta-learner — whose inputs are the base models' predictions and whose job is to learn how to combine them.

The trick that makes this work without cheating is generating each base model's predictions out-of-fold: each training example gets a prediction from a base model that never saw that example during its own training (typically via k-fold cross-validation), so the meta-learner trains on predictions that look like genuine out-of-sample forecasts rather than predictions the base models have already memorized.

Worked example. Three base models predict tomorrow's return: a linear model, a random forest, and a neural net. Instead of averaging their three predictions equally, a stacking setup trains a small logistic or linear meta-model on out-of-fold predictions from all three, and might learn that the random forest is reliable in calm markets but the neural net predicts better during high-volatility stretches — assigning weights like 0.5/0.2/0.3 that shift with a volatility feature included as a meta-model input, rather than a single fixed 0.33/0.33/0.33 split.