LSTM Gate Mechanics: Forget, Input, Output

A plain recurrent network's hidden state gets completely rewritten at every timestep by a $\tanh$ nonlinearity, and Backpropagation Through Time showed why that's fatal for long-range memory: the gradient has to pass through that rewriting operation at every single step, and each pass shrinks it, so information from 30 steps ago is, for training purposes, gone. What's needed is a second pathway — one where information can survive many timesteps essentially untouched unless something specifically decides to change it. The LSTM's answer is to add exactly that: a cell state that flows forward mostly by simple addition and multiplication by numbers close to 1, protected by three small learned "gates" that control what's allowed to touch it.

The analogy: a conveyor belt with three checkpoints

Picture a factory conveyor belt running the length of the building, carrying boxes (information) from one end to the other, largely undisturbed — nothing forces a box off the belt just because it passed a workstation. At each workstation there are three controls. A forget gate decides whether to remove specific boxes already on the belt (this item is now irrelevant, discard it). An input gate decides whether to place new boxes onto the belt (this new information is worth keeping). An output gate decides what to actually read off the belt right now and hand to the worker at this station, without necessarily removing it from the belt. The belt itself — the cell state — is the long-term memory; the three gates are the only things allowed to modify what's on it, and by default, absent a gate saying otherwise, a box just keeps riding along unchanged.

The maths

At each timestep, an LSTM computes three gates and two candidate updates, all from the same two inputs: the previous hidden state $h_{t-1}$ and the current input $x_t$. Each gate is a sigmoid — a function squashing its input to between 0 and 1, read as "how much to let through," where 0 blocks completely and 1 lets everything through:

$$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f) \quad \text{(forget gate)}$$ $$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i) \quad \text{(input gate)}$$ $$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o) \quad \text{(output gate)}$$

In words: three separate small networks, each looking at the same combined input (previous hidden state concatenated with current input), each outputting a number between 0 and 1 for every dimension of the cell state, saying how open that particular gate is right now. A candidate new memory content is also computed:

$$\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)$$

In words: what could be added to memory this step, before the input gate decides how much of it actually gets in.

The cell state update is the mechanism that matters most:

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$$

In words ($\odot$ means multiply element by element): keep whatever fraction of the old memory $C_{t-1}$ the forget gate says to keep, and add whatever fraction of the new candidate $\tilde{C}_t$ the input gate says to admit. If $f_t \approx 1$ and $i_t \approx 0$ at some position, that slot of memory passes through completely unchanged — no multiplication by a shrinking derivative, no rewriting, just addition of near-zero. That additive, gated pathway is exactly what a plain RNN's $h_t = \tanh(\dots)$ update lacks, and it's why the LSTM's gradient along the cell-state path doesn't vanish the way BPTT's does through a plain hidden state.

Finally the hidden state — what gets exposed to the next layer or the next timestep's other computations — is a filtered read of the cell state:

$$h_t = o_t \odot \tanh(C_t)$$

In words: squash the current memory through $\tanh$ to bound it, then let the output gate decide how much of it to actually reveal right now.

Worked example 1: one timestep, all three gates, real numbers

Simplify to a single memory dimension. Suppose at this timestep the gates, computed from $h_{t-1}$ and $x_t$, come out to $f_t = 0.9$, $i_t = 0.3$, $o_t = 0.7$, the candidate is $\tilde C_t = 0.8$, and the previous cell state is $C_{t-1} = 2.0$.

Cell state update:

$$C_t = (0.9)(2.0) + (0.3)(0.8) = 1.8 + 0.24 = 2.04$$

The forget gate kept 90% of the old memory ($1.8$ of the original $2.0$) and the input gate admitted 30% of the new candidate ($0.24$), giving a new memory of 2.04 — barely changed from 2.0, meaning this step mostly preserved existing memory with a small nudge.

Hidden state (what actually gets exposed):

$$h_t = 0.7 \times \tanh(2.04) = 0.7 \times 0.9668 \approx 0.677$$

Even though the memory itself barely moved, the exposed hidden state depends heavily on the output gate — here, only 70% of the squashed memory is revealed. A different output gate value, say $o_t = 0.1$, would give $h_t \approx 0.097$ from the exact same underlying memory, showing that "what's remembered" and "what's currently shown" are genuinely separate quantities.

Worked example 2: why the forget gate saves long-range memory

Compare two scenarios over 20 steps, tracking a single memory slot that should hold a signal from step 1 (say, "today opened with an unusual gap") until it's needed at step 20.

Plain RNN-like decay (no gating, roughly what a vanilla RNN's hidden state does): each step multiplies the value by something like 0.6 through the $\tanh$/weight combination.

$$0.6^{20} \approx 0.0000366$$

The original signal is essentially gone by step 20.

LSTM with a forget gate learned to stay near 1 for this feature, say $f_t = 0.98$ at every step, with the input gate keeping new noise from swamping it ($i_t \approx 0$ once the value is set):

$$0.98^{20} \approx 0.668$$

Roughly 67% of the original signal survives 20 steps — about 18,000 times more than the ungated case. The forget gate is a learned parameter, not fixed at 0.98; the network discovers, from data, that keeping this gate close to 1 for signals that matter over the long run minimizes loss, and gradient descent can find that setting precisely because the additive cell-state path keeps the gradient alive long enough to discover it.

Every gate is a sigmoid — the same S-shaped curve. Drag its steepness and note it's always bounded between 0 and 1: that boundedness is what makes "0.98 held over 20 steps" a stable, learnable target, unlike an unbounded value that could drift.

What this means in practice

LSTMs (and their simpler cousin, the GRU) remain a strong default anywhere a sequence has dependencies spanning dozens to a few hundred steps and the dataset is too small to make a full transformer's attention worth its cost — think tick-level microstructure features, a rolling window of daily factor exposures, or any series where a plain RNN's forgetfulness was the actual bottleneck. Each gate is itself a trainable set of weights, so an LSTM roughly quadruples the parameter count of a plain RNN cell of the same hidden size (four weight matrices instead of one) — a real cost worth weighing against a shorter-memory but cheaper alternative when sequence length is genuinely short.