Stacked and Deep Recurrent Networks

A single recurrent layer already turns a raw sequence into a sequence of hidden states summarizing what came before at each point. But those hidden states are still built from raw, low-level input — can a network build a second, higher-level summary on top of the first one's summaries, the way a deep feedforward network builds progressively abstract features layer by layer?

The analogy: a story passed through several editors

Imagine a manuscript passed through a chain of editors: the first editor reads the raw text sentence by sentence and jots down basic facts; the second editor doesn't reread the original text at all — they only read the first editor's running notes, sentence by sentence, and extract higher-level themes from those. Stacking recurrent layers works the same way: layer 2's input sequence is layer 1's output sequence of hidden states, not the original raw input.

The mechanics

Layer 1 processes the raw sequence $x_1, \dots, x_T$, producing hidden states $h_1^{(1)}, \dots, h_T^{(1)}$. Layer 2 treats that entire sequence of hidden states as its own input sequence:

$$h_t^{(2)} = \text{RNN}^{(2)}\big(h_t^{(1)}, h_{t-1}^{(2)}\big)$$

In words: at every timestep, the second layer's recurrence takes the first layer's output at that same timestep as its input, while also carrying forward its own recurrence across time — so information now flows both across time (within a layer) and across depth (between layers) before reaching the final output.

Worked example 1: two layers, three timesteps

Toy scalar RNN with weight $0.5$ on the recurrent connection and $1$ on the input, layer 1 processes raw inputs $x_1=2, x_2=1, x_3=3$: $h_1^{(1)}=0.5(0)+2=2$; $h_2^{(1)}=0.5(2)+1=2$; $h_3^{(1)}=0.5(2)+3=4$. Layer 2 now takes $(2,2,4)$ as its input sequence, with its own weights ($0.5$ recurrent, $1$ input): $h_1^{(2)}=0.5(0)+2=2$; $h_2^{(2)}=0.5(2)+2=3$; $h_3^{(2)}=0.5(3)+4=5.5$. Layer 2's final state, $5.5$, reflects a transformation of layer 1's already-processed summaries, not the raw $2,1,3$ directly.

Worked example 2: parameter and gradient-path growth

A single layer with hidden size 32 over a 50-step sequence has one recurrent weight matrix and a 50-step gradient chain. Stacking to 3 layers of the same hidden size triples the number of recurrent weight matrices, and a gradient flowing from the final loss back to an early input at layer 1 must now pass through 50 timesteps and 3 layers of nonlinearities — a much longer multiplicative chain, which is exactly where vanishing or exploding gradients become more likely.

Depth and dataset size trade off here the same way model complexity does more generally — watch how the gap between training and held-out error changes as you add capacity:

What this means in practice

Adding depth typically helps on complex sequence tasks with enough data, since each layer can build a more abstract temporal representation than the one below it. But because gradients must now survive both the time dimension and the depth dimension, deep recurrent stacks usually need residual or skip connections, layer normalization, and careful initialization to train reliably — the same tools used to tame depth in feedforward networks, applied here for a compounded reason.