Dropout Regularization

A large neural network has enough capacity to memorize its training data outright — including the noise. Left unchecked it will happily wire up a handful of units to detect quirks of specific training examples rather than genuine patterns, the way a research team that always works together starts covering for each other's blind spots instead of each person independently getting the analysis right. The team looks great on the deals they've already seen and falls apart on a new one.

Dropout breaks up that team on purpose. On every training pass, each hidden unit is temporarily deleted — its output forced to zero — with some fixed probability, independently of every other unit. A different random subset of the network trains on every batch, so no unit can count on any particular teammate being present. Each one has to become independently useful.

The mechanics, one symbol at a time

Let $h$ be a layer's output vector before dropout, and let $p$ be the keep probability (the chance any given unit survives this pass). Draw a random mask $r_i$ for each unit $i$, where $r_i = 1$ with probability $p$ and $r_i = 0$ otherwise:

$$\tilde{h}_i = r_i \cdot h_i, \qquad r_i \sim \text{Bernoulli}(p)$$

In words: flip an independent, biased coin for every unit on every pass; if it comes up "drop," that unit's output is zeroed for this pass only, as if it did not exist. At test time you want the network's full capacity, so no units are dropped — but their outputs must then be scaled by $p$ (or equivalently, scaled by $1/p$ during training, the more common convention), because at training time each surviving unit was implicitly compensating for its missing neighbors and would otherwise produce outputs that are too large on average.

Worked example 1: one forward pass with a fixed mask

A layer with four units produces raw outputs $h = (3, -1, 5, 2)$. Keep probability $p = 0.5$, and this pass's coin flips give the mask $r = (1, 0, 1, 0)$.

Masked output: $\tilde{h} = (3 \times 1,\ -1 \times 0,\ 5 \times 1,\ 2 \times 0) = (3, 0, 5, 0)$.

Using the "scale during training" convention, divide the surviving units by $p$: $\tilde{h} = (3/0.5, 0, 5/0.5, 0) = (6, 0, 10, 0)$. The next layer sees a network that, this batch, only has units 1 and 3 to work with, boosted so their expected contribution matches what the full network would have produced.

Run the next batch and a different mask might be $(0, 1, 1, 1)$ — a completely different sub-network trains on that pass. Over thousands of batches, the model has effectively trained something close to an enormous ensemble of overlapping smaller networks, all sharing the same weights.

Worked example 2: why it stops units from co-adapting

Suppose unit A has learned to detect "earnings surprise" and unit B has learned to detect "earnings surprise, confirmed by unit A" — B has become a parasite riding on A rather than an independent detector. Without dropout, that costs nothing during training because A is always present. With dropout at $p = 0.5$, unit A is absent on roughly half of all passes. On those passes, B's parasitic signal is worthless — it contributes nothing useful to the prediction, and the loss punishes the network for it. Over many passes, gradient descent is pushed toward B learning something that works whether or not A happens to be present that pass — an independently useful feature rather than a redundant echo. That is precisely what "prevents co-adaptation" means in practice.

Drag the complexity slider in the explorer above: dropout is one of the standard tools for pulling a model back from the high-variance, overfit side of that curve without shrinking its raw capacity, by making the effective model simpler on every individual pass.

What this means in practice

Dropout is typically applied to fully connected layers, with $p$ around 0.5 for hidden layers and closer to 0.8–0.9 for inputs (you rarely want to delete raw features outright). It is usually turned off — or replaced with the milder recurrent-dropout variants — inside recurrent layers, because randomly deleting a unit at every timestep of a sequence destroys the very memory those units are meant to carry. In modern architectures that already have batch normalization and residual connections, dropout is used more sparingly than in the era it was introduced, since those other tools provide some regularizing effect on their own; stacking all three aggressively can under-fit.