HiPPO Memory for Long-Range Dependencies

A recurrent network trying to remember something that happened thousands of steps ago faces a basic compression problem: it must summarize an ever-growing history using a hidden state of fixed size, and naive recurrence tends to forget the distant past exponentially fast. HiPPO ("High-order Polynomial Projection Operators") addresses this by asking a precise question: given a fixed-size vector of coefficients, what is the mathematically optimal way to update it, at every new time step, so that it always represents the best possible approximation of the entire input history seen so far under some family of basis functions (commonly Legendre polynomials)?

The answer turns out to be a simple linear ordinary differential equation, $\frac{d}{dt}c(t) = A c(t) + B x(t)$, where $c(t)$ is the coefficient vector and $A$, $B$ are fixed matrices derived analytically from the choice of basis and a weighting scheme over how much to emphasize recent versus distant history. Because $A$ and $B$ are derived in closed form rather than learned from scratch, the resulting recurrence has provably good long-range memory built in from initialization, rather than hoping training discovers it.

This construction is the mathematical seed behind the S4 and later structured state-space sequence models, which discretize the same continuous-time equation and train it on long sequences — audio, genomics, long text — where ordinary recurrent networks and even attention-based transformers can struggle with the sheer length of the dependency.