Peephole Connections and LSTM Variants

A standard LSTM cell's three gates (input, forget, and output) decide what to remember or forget by looking at the current input and the previous hidden output, but crucially not at the cell's own internal memory (the cell state) directly — the gates only see the hidden state, which is itself a filtered version of that memory. Gers and Schmidhuber's 2000 peephole LSTM variant gives the gates a direct "peephole" connection into the raw cell state, letting each gate's decision depend on exactly what's currently stored in memory, not just on a filtered summary of it.

The motivation was precise timing: the authors showed that a standard LSTM struggles at tasks requiring it to count events or measure precise time intervals between them, because the gates can't see the raw accumulated count sitting in the cell state — they only see it after it's been passed through the output gate's filter. Adding peephole connections let the gates track and act on the exact stored value directly, substantially improving performance on such timing- and counting-sensitive sequence tasks.

Peephole LSTMs are one of several LSTM variants proposed over the years (others include coupling the input and forget gates, or the simpler GRU cell that merges cell and hidden state entirely); in practice, later large-scale comparisons found peephole connections give only a modest, inconsistent benefit on most typical sequence tasks, so plain LSTM and GRU cells remain far more commonly used, with peephole connections reserved for problems with genuinely precise timing structure.