Rotary Position Embeddings (RoPE)

Attention compares two word vectors with a dot product to decide how relevant one is to the other. But the raw dot product of two vectors knows nothing about where those words sit in the sentence — swap "the fund gained" and "gained the fund" and the same two vectors for "fund" and "gained" would produce the identical score, which is wrong, because word order changes meaning. A transformer needs some way to inject position into the comparison, and it needs that injection to survive being copied into every layer of a very deep network without exploding or degrading. RoPE solves this by rotating each word's vector by an angle proportional to its position, so position becomes baked into the geometry of the vector itself rather than added on as a separate number.

The analogy: clock hands instead of a printed timestamp

Imagine stamping every document with the literal time it was created, like "14:32:07," and having a clerk compare two documents by subtracting the timestamps. That works, but the clerk has to do arithmetic every time, and the numbers grow without bound as the day goes on. Now imagine instead giving each document a clock hand pointing at the time it was created. To find out how far apart two documents are in time, the clerk just looks at the angle between the two hands — no subtraction needed, and the angle stays bounded no matter how late in the day it gets, because it wraps around the clock face. RoPE gives every word vector a "clock hand" — it rotates the vector by an angle proportional to the word's position — so that comparing two words' vectors automatically reads off their relative position as an angle, with no separate bookkeeping.

The mechanism: rotating pairs of coordinates

Take a word's vector and split it into pairs of coordinates: $(x_1, x_2)$, $(x_3, x_4)$, and so on. RoPE rotates each pair by an angle $\theta$ that depends on the word's position $m$ in the sequence and which pair it is:

$$\begin{pmatrix} x_1' \\ x_2' \end{pmatrix} = \begin{pmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$$

Here $m$ is the position of the word in the sentence (word 1, word 2, ...), $\theta$ is a fixed angle assigned to this particular pair of coordinates (different pairs get different, geometrically spaced angles, so some pairs rotate fast and others slow), and the matrix is a standard 2D rotation — it turns the point $(x_1, x_2)$ by angle $m\theta$ around the origin without changing its length. In plain English: each pair of numbers in the vector gets spun around like a clock hand, by an amount that grows with how far into the sentence the word appears.

The payoff shows up when two rotated vectors are compared by dot product. For a query at position $m$ and a key at position $n$, the dot product of their rotated pairs depends only on $(m-n)\theta$ — the difference in positions — not on $m$ and $n$ individually. Two words three apart get exactly the same positional contribution to their score whether they're words 4 and 7 or words 400 and 403. That is the entire design goal: bake in relative position, discard absolute position, without ever computing a subtraction.

Drag the angle in the explorer below to see how a 2D rotation preserves a vector's length while changing its direction — that is exactly what each coordinate pair of a RoPE vector undergoes, just repeated across many pairs with different angles.

Worked example 1: rotating a single pair by hand

Take the coordinate pair $(x_1, x_2) = (1, 0)$ at position $m=2$, with angle $\theta = 30°$ for this pair, so the rotation angle is $m\theta = 60°$. Using $\cos 60° = 0.5$ and $\sin 60° = 0.866$:

$$x_1' = 0.5(1) - 0.866(0) = 0.5, \qquad x_2' = 0.866(1) + 0.5(0) = 0.866$$

The rotated pair is $(0.5, 0.866)$. Check the length: $\sqrt{0.5^2 + 0.866^2} = \sqrt{0.25+0.75} = 1.0$ — unchanged from the original length of $1$, exactly as a rotation should behave; only the direction moved.

Worked example 2: relative position survives, absolute position doesn't

Take two positions, $m=5$ and $n=2$, both with the same starting pair $(1,0)$ and the same per-pair angle $\theta=20°$. Position 5's rotated pair uses angle $100°$: $(\cos 100°, \sin 100°) = (-0.174, 0.985)$. Position 2's uses angle $40°$: $(\cos 40°, \sin 40°) = (0.766, 0.643)$.

Their dot product is $(-0.174)(0.766) + (0.985)(0.643) = -0.133 + 0.633 = 0.500$. Now compare positions 8 and 5 — the same gap of 3 — using angles $160°$ and $100°$: $(\cos160°,\sin160°)=(-0.940,0.342)$ and $(\cos100°,\sin100°)=(-0.174,0.985)$. Dot product: $(-0.940)(-0.174)+(0.342)(0.985) = 0.164+0.337=0.500$. Same gap, same dot product, even though the absolute positions were completely different — the calculation only ever "knew" a difference of $3\theta = 60°$.

What this means in practice

RoPE is the default positional scheme in most modern large language models because it plays well with the KV cache — a cached key rotated for position 50 is still valid once the sequence reaches position 5000, since nothing about how it was rotated changes, only which query it's compared against — and because relative-position encoding tends to generalize better to sequence lengths longer than training. It also underlies "context extension" tricks: by rescaling $\theta$, engineers can stretch a model trained on short sequences to handle much longer ones, in effect slowing down the clock hands.

Practice

1. Rotate the pair $(0, 1)$ by $90°$ by hand using the rotation matrix, and confirm the length is unchanged.
2. Why does the dot product between a rotated query and a rotated key depend only on $m - n$ and not on $m$ or $n$ separately? Point to the step in the derivation where the individual angles would cancel.
3. If a model was trained with sequences up to position 512 and $\theta$ is halved at inference, does a fixed gap of 10 positions now correspond to a larger or smaller rotation angle than during training?