GRU vs LSTM Tradeoffs

An LSTM regulates what it remembers with three separate gates — input, forget, output — plus a dedicated internal "notebook," the cell state, that those gates write to and read from. That design works well, but it is also relatively heavy: four full weight matrices per layer. A GRU asks whether a lighter design, without the separate notebook, can do nearly as well.

The analogy: two thermostat designs

An LSTM is a thermostat with three independent dials — how much old heat to keep, how much new heat to let in, how much of the result to actually show on the display — writing to a dedicated internal temperature log. A GRU is a simpler thermostat with two dials — how much to update, and how much of the old reading to reset — that blends old and new directly into one running reading, with no separate internal log to maintain.

The mechanics

LSTM's cell state update: $c_t = f_t \odot c_{t-1} + i_t \odot g_t$ — in words, keep a forget-gated fraction of the old cell memory, and add an input-gated fraction of new candidate information, with the hidden state $h_t$ then read out from $c_t$ through a separate output gate. GRU's hidden state update:

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde h_t$$

In words: the update gate $z_t$ directly decides the mixing proportion between the old hidden state and a new candidate $\tilde h_t$ — no separate cell state, no separate output gate, just one running state that is a weighted blend.

Worked example 1: a GRU update by hand

Update gate $z_t = 0.3$, old hidden state $h_{t-1}=2$, candidate $\tilde h_t = 6$: $h_t = (1-0.3)(2) + 0.3(6) = 0.7(2) + 0.3(6) = 1.4 + 1.8 = 3.2$. A low $z_t$ would keep the state close to $2$; here $z_t=0.3$ pulls it moderately toward the new candidate.

Worked example 2: an LSTM update and a parameter count

Forget gate $f_t=0.8$, old cell state $c_{t-1}=2$, input gate $i_t=0.4$, candidate $g_t=5$: $c_t = 0.8(2) + 0.4(5) = 1.6+2 = 3.6$. Now compare parameter counts for hidden size 64, input size 32: LSTM needs 4 weight matrices (input, forget, output, candidate) of combined input size $64+32=96$, giving roughly $4 \times [96 \times 64 + 64] \approx 24{,}832$ parameters per layer; GRU needs 3 such matrices (update, reset, candidate), giving roughly $3 \times [96 \times 64 + 64] \approx 18{,}624$ — about 25% fewer parameters for the same hidden size.

What this means in practice

With a tight data or compute budget — for example, an intraday signal model retrained frequently — GRU's leaner design often trains faster and generalizes at least as well as an LSTM of the same hidden size. LSTM's extra cell state and output gate can help on tasks needing finer control over what is retained internally versus exposed externally, and sometimes edges ahead on longer or more intricate dependencies, but the gap is usually small enough that both should be benchmarked rather than assumed.