Snapshot Ensembles of Neural Networks

Ensembling — averaging predictions from several separately trained models — reliably improves accuracy, but training many full networks from scratch is expensive. Snapshot ensembles get most of the benefit from a single training run by using a cyclic learning rate schedule: the learning rate is repeatedly annealed down to near zero (letting the network settle into a good local minimum) and then sharply raised again (kicking it out of that minimum to explore a different part of the loss landscape), and a snapshot of the model's weights is saved at the bottom of each cycle. Because each cycle's minimum is reached from a different point in weight space, the saved snapshots tend to make genuinely different errors on the same inputs, which is exactly the diversity an ensemble needs to average away individual mistakes, even though all snapshots came from one continuous training run at roughly the cost of training a single model.

At prediction time, the snapshots are simply averaged like any other ensemble, typically improving test accuracy over the single best snapshot alone, for a small fraction of the training cost that a true multi-model ensemble would require.

The number of cycles trades off diversity against per-snapshot quality: more, shorter cycles produce more snapshots but each one converges less fully, while fewer, longer cycles give better-trained individual snapshots but less diversity to average over — in practice a handful of cycles (roughly four to six) tends to strike a reasonable balance for most image and tabular models.