Learned Positional Embeddings

Sinusoidal positional encoding hand-designs a fixed pattern of waves to represent position, chosen by the engineers before training ever starts, on the theory that this particular mathematical pattern is a good one for a model to build on. But nothing guarantees the sine/cosine pattern is actually the best way to represent position for a given task — it's a reasonable, useful guess, not a proof of optimality. The alternative is to stop guessing: treat position the same way a token's identity is treated, as an index into a lookup table of vectors, and let the training process itself discover whatever positional representation minimizes the loss.

The analogy: assigned seats versus seats you personalize yourself

Sinusoidal encoding is like a venue that stamps every seat with a fixed, pre-designed pattern of colored lights before anyone arrives — reasonable and systematic, but decided in advance with no feedback from how people use the seats. Learned positional embeddings are like giving every seat its own blank nameplate that gets engraved gradually, based on how that seat is actually used — seat 1 might end up engraved very differently from seat 50, in whatever way experience shows is helpful, rather than a pattern chosen beforehand.

The mechanics: a trainable lookup table indexed by position

A learned positional embedding is a matrix $P \in \mathbb{R}^{L_{\max} \times d}$, where $L_{\max}$ is the maximum sequence length the model supports and $d$ is the embedding dimension — one full trainable row per position, exactly analogous to a token embedding table but indexed by position instead of word identity:

$$x_i = e_i + P_i$$

In words: the input to the model at position $i$ is the token's content embedding $e_i$ plus that position's learned vector $P_i$, and both $e$ and $P$ are ordinary trainable parameters updated by gradient descent along with every other weight in the network — there is no hand-designed formula involved, only initialization (usually random) and whatever the training objective pushes the vectors toward. Because $P$ has exactly $L_{\max}$ rows, a model trained this way has no defined behavior for any position beyond $L_{\max}$ — unlike the sinusoidal formula, which is defined for any position at all, a learned table simply has no row to look up past the length it was built for.

Worked example 1: what gets learned, roughly

Suppose a model is trained on movie reviews averaging 40 words, with $L_{\max} = 512$. Position vectors $P_1$ through $P_{40}$ get abundant gradient signal (nearly every example touches them), while $P_{200}$ through $P_{512}$ get almost none, since few training examples are that long. Those rarely-seen late positions often end up close to their random initialization, poorly trained, and a model processing a genuinely 500-word input at inference may perform noticeably worse there — a direct consequence of position embeddings being learned from data rather than fixed by formula.

Worked example 2: comparing parameter cost

A model with $d = 768$ and $L_{\max} = 512$ has a positional embedding table of $512 \times 768 = 393{,}216$ trainable parameters, scaling linearly with maximum length; doubling $L_{\max}$ to 1024 doubles the table to 786,432. Sinusoidal encoding, by contrast, costs zero trainable parameters regardless of $L_{\max}$, since it's computed by formula rather than stored — a real, if usually minor, tradeoff between the two approaches.

Parameter count in a learned table grows linearly with $L_{\max}$ (drag the exponent toward 1 on this curve to see that straight-line growth) — a cost sinusoidal encoding never pays, since it needs no stored table at all.

What this means in practice

Learned positional embeddings were the standard choice in models like BERT and GPT-2, and they can outperform sinusoidal encoding on the length range seen during training, since the model gets to specialize its position representation to the actual data distribution rather than a generic formula. Their inability to extrapolate cleanly past $L_{\max}$ is exactly the gap that motivated relative-position schemes like ALiBi and Relative Position Bias and Rotary Position Embeddings (RoPE), both of which aim to keep learned-style flexibility while still generalizing to longer sequences than seen in training.

Practice

1. For $d=512$, $L_{\max}=1024$, compute the number of parameters in the positional embedding table.
2. Explain why position embeddings for late positions in a long-max-length model can end up poorly trained.
3. Give one concrete reason a production system serving inputs of highly variable length might prefer sinusoidal encoding over a learned table.