Lion and Sign-Based Optimizers

Most popular optimizers, like Adam, track both a momentum term (a running average of past gradients) and a per-parameter scaling based on the size of past squared gradients, which means storing two extra numbers for every single weight in the model. Lion strips this down: it keeps only a momentum term, and instead of using its exact value it uses only its sign — plus or minus one — to decide the direction of each update. The step size then comes entirely from a fixed learning rate rather than any adaptive scaling.

Think of it like a hiker who, instead of measuring exactly how steep the slope is at each step, just checks "is it currently trending up or down" and takes a fixed-size step accordingly. That's cruder information, but it turns out to be surprisingly effective, and it needs far less memory to track, because there's no running record of squared gradients to store — only one momentum buffer instead of two.

A worked example

Training a model with a billion parameters, switching from Adam to Lion roughly halves the optimizer's own memory footprint, since Adam stores two extra numbers per parameter (momentum and squared-gradient average) while Lion stores only one. On models where this was tested, Lion also reached comparable or slightly better final accuracy than Adam despite discarding gradient magnitude information, though it is typically more sensitive to the choice of learning rate and needs a smaller one than Adam would use.