LSTM and GRU Networks

A plain recurrent network is supposed to carry information forward through time: read one observation, update a summary of everything seen so far, read the next, update again. In practice it forgets almost immediately. Feed it a hundred minutes of order-flow data and by minute twenty the trace of minute one has been multiplied away to nothing. LSTM and GRU cells are the engineering answer to that failure, and they remain the default when you want a sequence model that fits on a laptop.

The analogy: a notebook with valves

Picture a conveyor belt running left to right through the whole sequence. On the belt sits a small notebook holding whatever the model currently believes is worth remembering. At every time step the belt passes three valves:

• an erase valve, which decides how much of the existing note to rub out,
• a write valve, which decides how much of a freshly drafted note to add,
• a read valve, which decides how much of the resulting note to show to the rest of the network right now.

Nothing forces the belt to be rewritten. If the erase valve stays shut, the note travels untouched across a hundred steps. A plain recurrent network has no belt: it copies the note through a squashing function every step, and copying degrades. That single structural difference is the whole idea.

Putting numbers on the valves

Write $c_t$ for the note on the belt (the cell state) and $h_t$ for what the cell shows the outside world (the hidden state). At each step the network sees the new observation $x_t$ and the previous output $h_{t-1}$, and computes three valve openings:

$$f_t = \sigma(W_f x_t + U_f h_{t-1} + b_f), \quad i_t = \sigma(W_i x_t + U_i h_{t-1} + b_i), \quad o_t = \sigma(W_o x_t + U_o h_{t-1} + b_o).$$

In words: each valve is a small linear score of "what I just saw plus what I was thinking", squashed by the logistic function $\sigma$ into a number between 0 and 1 — zero fully shut, one fully open. The weights $W$, $U$ and biases $b$ are learned like any others.

It also drafts a candidate note, using $\tanh$ to keep the draft between $-1$ and $1$:

$$\tilde{c}_t = \tanh(W_c x_t + U_c h_{t-1} + b_c).$$

Then the two updates that matter:

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t).$$

In words: keep an $f_t$ fraction of the old note, add an $i_t$ fraction of the new draft, reveal an $o_t$ fraction of the result. The symbol $\odot$ means elementwise multiply — each memory slot gets its own independent valve.

Every valve is a logistic curve. Drag the steepness below and watch how fast it goes from "shut" to "open" — that sharpness is what lets a gate behave like a switch rather than a dimmer.

Worked example: one step, by hand

Take one memory slot carrying $c_{t-1} = 0.80$, meaning "a strong uptrend is in force". A new bar arrives and the linear layers produce $z_f = 2.2$, $z_i = -0.85$, $z_c = 1.0$, $z_o = 0.4$. Squash each:

• $f_t = \sigma(2.2) = 1/(1 + e^{-2.2}) = 1/1.111 = 0.900$
• $i_t = \sigma(-0.85) = 1/(1 + e^{0.85}) = 1/3.340 = 0.299$
• $\tilde{c}_t = \tanh(1.0) = 0.762$
• $o_t = \sigma(0.4) = 0.599$

Now the belt update:

$$c_t = 0.900 \times 0.80 + 0.299 \times 0.762 = 0.720 + 0.228 = 0.948.$$

The cell kept 90 percent of what it had and topped it up. The output:

$$h_t = 0.599 \times \tanh(0.948) = 0.599 \times 0.739 = 0.443.$$

It believes the trend slightly more than before, but exposes only about 44 percent of that belief downstream. Now suppose the next bar looks like a regime break and drives the erase score to $z_f = -2.2$, so $f_t = 0.100$. The same arithmetic gives $c_t = 0.100 \times 0.80 + 0.228 = 0.308$: the old belief is almost entirely wiped in one step. One learned number decides between "hold for a hundred bars" and "forget everything now".

Worked example: how much gradient survives

Here is why gates fix training. In a plain recurrent network the gradient flowing back through $k$ steps gets multiplied by roughly the same recurrent weight each time. Take that factor to be $0.6$:

$$0.6^{10} = 0.0060, \qquad 0.6^{20} = 3.7 \times 10^{-5}, \qquad 0.6^{50} = 8.1 \times 10^{-12}.$$

After fifty steps the learning signal has been divided by a hundred billion — indistinguishable from zero in 32-bit arithmetic. Its half-life is $\ln(0.5)/\ln(0.6) = 1.4$ steps.

Along an LSTM cell state the multiplier is the erase gate instead. With $f_t = 0.95$:

$$0.95^{10} = 0.599, \qquad 0.95^{20} = 0.358, \qquad 0.95^{50} = 0.077.$$

Nearly eight percent survives fifty steps — a half-life of 13.5 steps, and about ten billion times more gradient than the plain network. Nothing exotic happened: 0.95 is simply much closer to 1 than 0.6 is, and the network can learn to keep it there.

The GRU: the same idea with fewer parts

The gated recurrent unit merges the erase and write valves. Whatever it does not keep, it writes:

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t.$$

In words: $z_t$ is a single dial between "keep the old state" and "take the new one", and the two sides always sum to one. A second valve, the reset gate $r_t$, controls how much of the old state may influence the draft $\tilde{h}_t$. There is no separate cell state; $h$ does both jobs.

That is three weight blocks instead of four — roughly 25 percent fewer parameters and a faster step. On financial sequences, which are short, noisy and data-poor, the GRU usually matches the LSTM and sometimes beats it precisely because it has less to overfit.

What this means in practice

On a desk these cells appear wherever you need a state rather than a snapshot: a volatility regime that persists, an inventory that accumulates, an order-book imbalance that builds. Three practical points matter more than the architecture choice:

• Initialise the forget bias positive. Setting $b_f \approx 1$ starts every gate near $\sigma(1) = 0.73$, so the cell remembers by default and learns to forget. Starting at zero starts training from amnesia, and it often never recovers.
• Clip the gradients. Gates fix vanishing but not exploding — see Vanishing and Exploding Gradients. A global-norm clip at 1 costs nothing and stops one outlier bar wrecking the weights.
• Respect the time ordering. Shuffling whole sequences is fine; shuffling within one is not, and neither is a scaler fitted on the full history. Standardise on past data only, or you have built a look-ahead.

Gated cells remain the sensible default at small data scale. Their one structural limit is that information still travels step by step along the belt — exactly the constraint the transformer removes by letting every position look at every other in a single hop.