Greedy Ensemble Selection with Replacement

After a large hyperparameter search or AutoML run, you might end up with hundreds of trained models — different algorithms, different settings — sitting in a library. Greedy ensemble selection builds a final ensemble from that library not by picking one winner, but by repeatedly adding whichever model, when averaged into the current ensemble, most improves performance on a validation set — and critically, the same model can be added more than once, effectively giving it more weight.

Worked example. A library holds 200 trained classifiers. The selection process starts by picking the single best-performing model as the initial ensemble. At step two, it tries averaging in each of the other 199 models one at a time and keeps whichever addition raises validation AUC the most — say, a random forest. At step three, it repeats the same test, including trying the random forest again, since replacement is allowed; if re-adding it (giving it double weight) improves validation performance more than adding any new model, it gets picked again. This continues for a fixed number of rounds or until validation performance stops improving.

Because it optimizes directly against a validation metric rather than picking models by a proxy like individual accuracy, greedy selection tends to favor models that are diverse and complementary, not just individually strong — a weaker model that makes different mistakes can still earn a slot if it corrects the ensemble's blind spots.