Legendre Memory Units

Ordinary recurrent cells like LSTMs learn their memory dynamics entirely from data, and that memory tends to fade over long sequences because nothing forces the learned recurrence to preserve distant history well. The Legendre Memory Unit (LMU) takes a different approach: instead of learning the recurrence dynamics from scratch, it hard-wires them to a closed-form solution — the same HiPPO derivation that produces the mathematically optimal linear ODE for compressing a continuous history into a fixed-size vector of coefficients under a Legendre polynomial basis.

Concretely, an LMU cell maintains a memory vector $m$ updated by a fixed, non-learned linear recurrence (the discretized HiPPO-Legendre matrices $A$ and $B$), which is then fed into standard learnable nonlinear layers to produce the cell's hidden state and output, much like an LSTM's gates operate on top of its cell state. Because the memory update itself is fixed by a mathematically derived formula rather than learned, the LMU needs very few parameters to represent long time windows accurately, and it demonstrably preserves information over thousands of time steps — well beyond where standard recurrent cells begin to degrade — while using far fewer parameters to do so.

LMUs have been used for tasks needing genuine long-range memory with tight parameter budgets, such as embedded or low-power sequence processing, and they served as an early proof-of-concept that HiPPO-style fixed linear memory could outperform fully learned recurrence, a lineage that continues in later structured state-space sequence models.