ALiBi and Relative Position Bias

Both sinusoidal and learned positional encoding share a weakness: they inject position information into the token embeddings at the very start, before any attention happens, and then hope the attention mechanism learns to make good use of that signal on its own. Learned embeddings additionally break down entirely past whatever maximum length they were trained on, since there's simply no table entry for a position beyond it. ALiBi takes a different approach entirely — don't touch the embeddings at all, and instead bias the attention scores themselves, directly and explicitly, by how far apart two tokens are.

The analogy: a conversation that naturally favors recent context

Think of a group discussion where, without anyone saying so explicitly, people's attention naturally leans toward whatever was said most recently — not because older comments are irrelevant, but because relevance decays gradually with how long ago something was said, and everyone applies roughly the same discount rate to older statements. ALiBi builds exactly this discount directly into attention: rather than hoping the model discovers "nearby things usually matter more" from a positional embedding pattern, it hard-wires a distance-based penalty straight into every attention score, so the preference for nearby context is guaranteed by construction, not something that has to be learned from scratch.

The mechanics: a linear penalty added straight to attention scores

ALiBi (Attention with Linear Biases) adds a penalty proportional to the distance between query position $i$ and key position $j$ directly to the raw attention score, before softmax:

$$\text{score}(i,j) = \frac{q_i \cdot k_j}{\sqrt{d}} - m \cdot |i - j|$$

In words: take the ordinary scaled dot-product score exactly as before, then subtract a penalty equal to a fixed slope $m$ times the distance between the two positions — the further apart two tokens are, the larger the subtraction, so distant tokens are steered toward lower attention weight after softmax, with no positional information added to the embeddings at all. The slope $m$ is set differently per attention head (following a fixed geometric schedule decided in advance, not learned), so different heads end up favoring different effective ranges: some heads with a steep slope focus almost entirely on nearby tokens, others with a shallow slope stay more evenly spread across long distances.

Because this penalty is defined for any distance $|i-j|$, including distances far larger than anything seen during training, ALiBi extrapolates to much longer sequences at inference time than it was trained on — a model trained on 1,024-token sequences can be run on 4,096-token sequences with only a modest, predictable quality change, something neither sinusoidal nor learned embeddings reliably achieve.

Worked example 1: computing a penalized score by hand

Suppose the raw scaled score between query position 10 and key position 7 is $2.0$, and this head has slope $m = 0.25$. Distance is $|10-7|=3$, penalty $0.25 \times 3 = 0.75$, biased score $2.0 - 0.75 = 1.25$. Compare key position 2: raw score also $2.0$, distance $|10-2|=8$, penalty $0.25 \times 8 = 2.0$, biased score $2.0 - 2.0 = 0$. Even though both had identical raw relevance, the more distant position ends up with a lower biased score purely from being further away — the "recency discount," applied uniformly and predictably.

Worked example 2: different heads, different effective ranges

Say one head has slope $m=0.5$ (steep) and another $m=0.03$ (shallow), both looking at tokens 20 positions apart with identical raw score $3.0$. Steep head: penalty $= 0.5 \times 20 = 10$, biased score $= 3.0 - 10 = -7$ — after softmax against nearer competitors, this pair contributes almost nothing. Shallow head: penalty $= 0.03 \times 20 = 0.6$, biased score $= 2.4$ — still highly competitive. The steep head behaves like a short-range local-attention head, the shallow head like a long-range one, purely from the fixed per-head slope, with no extra learned parameters needed.

Picture the penalty term $m \cdot |i-j|$ as a straight line rising with distance (a simple linear, not curved, function) — it's subtracted directly from the score, so bias grows smoothly and predictably the further apart two positions are, with no discontinuity at any particular length.

What this means in practice

ALiBi is used specifically where a model needs to reliably handle sequences longer at inference than anything seen during training — long-document processing, extended chat histories, or any deployment where input length can't be tightly bounded in advance. It trades away the fully-learned flexibility of learned embeddings for a built-in, distance-based inductive bias that turns out to generalize far better, and it adds essentially no extra parameters or compute relative to ordinary attention, since it's just one subtraction per score.

Practice

1. Compute the ALiBi-biased score for raw score $1.5$, distance $6$, slope $m=0.2$.
2. Explain why a head with a very small slope $m$ behaves more like a "look anywhere" head than a "look nearby" head.
3. Why can ALiBi be applied to sequence lengths never seen during training, unlike a learned positional embedding table?