Sinusoidal Positional Encoding

Self-attention computes a weighted average of value vectors based on how similar queries and keys are — and that computation is completely blind to order. If you shuffled the tokens of a sentence and shuffled the attention weights identically to match, the output would be unchanged: attention treats a sequence as a bag of vectors, not an ordered list. But word order obviously matters — "dog bites man" and "man bites dog" contain the same words in a different order with a very different meaning — so a transformer needs some way to smuggle position information into a mechanism that otherwise has no concept of it at all.

The analogy: giving every seat a unique, decodable fingerprint

Imagine a lecture hall where every seat looks identical, and a mechanism (attention) that only ever asks "how similar is this person's interests to that person's," with no way to ask "which seat are they in." If you wanted position to matter anyway, you could stamp each seat with a unique combination of colored lights — some blinking fast, some slow, some barely at all — such that no two seats ever show the same combination, and nearby seats show similar (not identical) patterns. Add that pattern to each person's badge before they interact, and interactions can now be influenced by relative seat position, without the interaction mechanism needing to understand "seats" at all.

The mechanics: one wave per dimension, mixing frequencies

For position $pos$ and embedding dimension index $i$ (out of $d$ total dimensions), the sinusoidal positional encoding is:

$$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$$

In words: each pair of dimensions gets a sine and cosine wave at a different frequency, changing smoothly across dimensions — low indices oscillate fast (short wavelength), high indices oscillate slowly (long wavelength). This vector, same size as the token embedding, is simply added to it before the first attention layer, so every token's representation carries both "what word this is" and "where it sits."

The key useful property: because $\sin(a+b) = \sin a \cos b + \cos a \sin b$, the encoding at position $pos + k$ can be written as a fixed linear function of the encoding at $pos$, for any offset $k$. In words: the relationship between two positions a fixed distance apart is consistent regardless of where in the sequence that pair sits — position 5 and 8 relate the same way position 50 and 53 do, helping the model generalize to relative distances at new absolute positions.

Worked example 1: computing two dimensions by hand

Take $d = 4$, position $pos = 1$. For $i=0$: $PE_{(1,0)} = \sin(1) = 0.841$, $PE_{(1,1)} = \cos(1) = 0.540$. For $i=1$: $PE_{(1,2)} = \sin(1/100) = 0.010$, $PE_{(1,3)} = \cos(0.01) = 1.000$. Dimension pair 0 (fast wave) has already moved substantially by position 1, while pair 1 (slow wave) has barely moved — giving each position a genuinely unique combined fingerprint even though any single pair alone would eventually repeat.

Worked example 2: distinguishing nearby versus distant positions

Compare positions 1 and 2 on the fast pair: $\sin(1)=0.841$ versus $\sin(2)=0.909$ — a modest change. Compare positions 1 and 500 on that same fast dimension: $\sin(1)=0.841$ versus $\sin(500)=-0.262$ — a large change, since the fast wave has cycled many times by position 500. The slow pair, meanwhile, barely distinguishes 1 and 2, but clearly distinguishes 1 and 500. Combining many frequencies means nearby positions get similar encodings on most dimensions (helpful for local patterns) while distant positions still end up clearly distinguishable overall.

Each dimension pair contributes one wave at its own frequency; imagine several curves like this one stacked at different speeds — no single wave alone uniquely identifies a position, but the combination across all of them does, the same way a chord is unique even though individual notes repeat.

What this means in practice

Sinusoidal encoding was the original transformer's choice because it needs no learned parameters and can, in principle, extrapolate to sequence lengths longer than anything seen in training, since the formula is defined for any position — a property that later, more flexible alternatives like Learned Positional Embeddings and ALiBi and Relative Position Bias each trade off differently. It's a fixed, deterministic pattern added once at the input, not recomputed at every layer.

Practice

1. Compute $PE_{(0, 0)}$ and $PE_{(0, 1)}$ for any $d$ — what value do all dimension-0 sine terms take at position 0, and why?
2. Explain why using only a single frequency (one sine wave) for positional encoding would fail to uniquely identify positions in a long sequence.
3. Why does the addition-formula property help a transformer generalize to relative offsets it has seen before, even at new absolute positions?