Pasting, Random Subspaces and Random Patches

Bagging trains each model in an ensemble on a bootstrap sample — rows drawn with replacement from the training set. Three close relatives vary the recipe for how each model's training data is sliced, trading off diversity, speed, and how well each variant suits high-dimensional data.

The three variants

Pasting draws a subset of the training rows for each model, but without replacement, so no row is repeated within a single model's sample (unlike bagging, where a row can appear multiple times). It's mainly useful when the full dataset is too large to fit in memory at once — each model just sees a smaller, disjoint-ish chunk.

Random subspaces keeps every row but randomly selects a subset of the features (columns) for each model to train on, which is especially useful when the number of features is very large relative to the number of rows, since it forces each model to rely on different, possibly less obvious signals.

Random patches combines both ideas at once: each model gets a random subset of rows and a random subset of columns, maximizing diversity between ensemble members at the cost of each individual model seeing less data.

Worked example

Training an ensemble of 100 trees on a dataset with 10,000 rows and 500 features: bagging would draw 10,000 rows with replacement (some rows repeated) using all 500 features per tree; random subspaces would use all 10,000 rows but only, say, 50 randomly chosen features per tree; random patches might use 5,000 rows and 50 features per tree, giving each of the 100 trees a genuinely different, overlapping-but-not-identical view of the data.