Stochastic Depth and Layer Dropping

Very deep residual networks (hundreds of layers) are slow to train and can overfit or suffer vanishing gradients, even though residual connections already let information skip past any single layer. Stochastic depth takes that skipping further: during training, each layer is randomly dropped entirely with some probability $p$ for that training step, so the input passes straight through the residual shortcut with no transformation at all, while at test time every layer is kept but its output is scaled by $(1-p)$ to match the average behavior seen during training. This is dropout applied to whole layers instead of individual neurons, and it is usually applied with a linearly increasing drop probability from the first layer (rarely dropped) to the last (dropped more often), since early layers extract more essential low-level features.

For example, a 110-layer residual network trained with stochastic depth (survival probability 0.5 at the deepest layer) trains noticeably faster per step, since roughly half the layers are skipped on any given pass, and it also generalizes better on held-out data than the same network trained with every layer always active, because the network is forced to learn representations that don't depend on any one layer being present. The technique also directly shortens the effective path gradients travel during backpropagation on any given step, which helps mitigate vanishing gradients in networks with hundreds of layers — a problem residual connections reduce but don't fully eliminate on their own.

The linear decay in drop probability by depth matters in practice: dropping an early layer that computes basic edge- or trend-like features tends to hurt more than dropping a late layer refining an already well-formed representation, so keeping early layers close to always-on while letting later ones be skipped aggressively gets most of the speed and regularization benefit with the least accuracy cost.