Recurrent and Variational Dropout

Ordinary dropout randomly silences a fresh subset of neurons on every single forward pass, and that works well between independent layers of a feedforward network. But a recurrent network's hidden state is deliberately built to carry information forward through time — so what happens when you apply dropout, unmodified, to the very connections responsible for that carrying?

The analogy: plugging your ears mid-sentence

Ordinary per-step dropout on a recurrent connection is like randomly plugging your ears for a different word at every syllable of a sentence — you'd never catch a coherent thread, because whatever you managed to retain gets randomly discarded again almost immediately. Variational (or "locked") dropout instead picks one random subset of connections to silence for the entire sentence, so a signal path that survives is either present throughout or absent throughout, and only a fresh random choice is made for the next sentence.

The mechanics

Naive dropout resamples a fresh mask at each timestep: $\tilde h_t = h_t \odot m_t$, with $m_t \sim \text{Bernoulli}(1-p)$ drawn independently per step. Variational dropout instead samples one mask per sequence and reuses it at every step:

$$\tilde h_t = h_t \odot m, \qquad m \sim \text{Bernoulli}(1-p) \text{ fixed for all } t$$

In words: the same units are dropped at every timestep within one sequence, so any unit that survives the mask stays available for the recurrence to actually use across time — it's only the choice of which units get to participate that changes from one training sequence to the next, never from one timestep to the next within a sequence.

Worked example 1: locked versus resampled masks

Hidden size 4, four timesteps, dropout probability $p=0.25$ (keep probability $0.75$). Variational dropout draws one mask, say $m=(1,0,1,1)$, and reuses it at every one of the four steps — unit 2 is silenced throughout, the other three are always present. Naive dropout instead draws four separate masks, one per step, e.g. $(1,0,1,1)$, $(0,1,1,0)$, $(1,1,0,1)$, $(0,0,1,1)$ — a unit alive at step 1 is quite likely dead at step 2, so anything it was carrying forward through the recurrence gets randomly erased almost immediately.

Worked example 2: survival probability across the sequence

With keep probability $1-p=0.7$ and a 4-step sequence, variational dropout keeps a given unit available at all four steps with probability exactly $0.7$, since it's the same mask throughout. Naive dropout requires the same unit to independently survive four separate draws: $0.7^4 = 0.24$ — meaning only about a quarter of units that were "alive" at step 1 remain alive by step 4, versus 70% under variational dropout. That gap is exactly why naive dropout on recurrent connections can erode memory far more aggressively than intended.

The naive case's survival probability decays exponentially with sequence length — watch how quickly the curve collapses as the exponent grows:

What this means in practice

Standard production RNN and LSTM implementations (for example, PyTorch's recurrent-dropout options) apply the locked, variational form specifically to the recurrent (hidden-to-hidden) connections. Ordinary per-step dropout remains fine on the non-recurrent connections — the input-to-hidden or hidden-to-output projections — where there is no time-carried state to protect.