Causal and Padding Attention Masks

Plain self-attention lets every position attend to every other position, with no restriction. That's correct when the whole sequence is genuinely available at once — but two common situations break that assumption. First, a model trained to generate text one token at a time must never see tokens after the one it's currently predicting, or it would learn to "cheat" by looking ahead, then be useless at actual generation, where the future genuinely doesn't exist yet. Second, when training on a batch of different-length sequences, shorter ones get padded with filler tokens to match the longest — and without intervention, real tokens would compute attention scores against that meaningless filler as if it were real content. Both problems are solved the same way: force certain attention scores to be ignored before softmax runs.

The analogy: an exam with a shutter, and empty seats

A causal mask is like an exam room fitted with a shutter sliding forward with each question, so a student answering question 5 can see questions 1–5 but the shutter physically blocks 6–10 from view — no amount of trying lets them peek ahead, because the information isn't accessible yet. A padding mask is different: imagine a classroom with empty desks marked "no student here." A discussion would never call on an empty desk expecting an answer — you'd just skip it. Padding masks make a model do the analogous thing: skip padding "seats" entirely.

The mechanics: additive masking before softmax

Both mask types work by adding a very large negative number to the attention scores at forbidden positions, before the softmax step, so that after softmax those positions get essentially zero weight:

$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d}} + M\right) V$$

In words: compute ordinary scaled attention scores exactly as usual, then add a mask matrix $M$ that is $0$ at every allowed position and $-\infty$ (in practice, a very large negative number like $-10^9$) at every forbidden position, before the softmax turns scores into weights. Because $\exp(-\infty) = 0$, forbidden positions get an attention weight of essentially zero and contribute nothing to the output, without changing the shape of the computation or requiring any special-case code inside the softmax itself.

A causal mask sets $M_{i,j} = -\infty$ for every $j > i$ (every key position after the current query position), giving a lower-triangular pattern of allowed positions. A padding mask sets $M_{i,j} = -\infty$ for every $j$ that is a padding token, regardless of $i$, blocking attention to filler columns entirely; the two masks are simply added together when both apply, as they typically do when training a decoder on a padded batch of variable-length sequences.

Worked example 1: causal masking on a 4-token sequence

In a 4-token sequence (0-indexed), row 1 of the causal mask is $M_1 = [0, 0, -\infty, -\infty]$: position 1 can see positions 0 and 1, but is blocked from 2 and 3. Suppose raw scores at row 1 are $[1.0, 0.5, 3.0, 2.0]$; after adding the mask they become $[1.0, 0.5, -\infty, -\infty]$, and softmax gives weights $[0.62, 0.38, 0, 0]$ — positions 2 and 3 get exactly zero weight, however large their raw score was, because the mask overrides it before softmax ever runs.

Worked example 2: padding masking in a batch

A batch has "the cat sat" (3 real tokens) and "the dog ran home fast" (5 real tokens), padded to a common length of 5, so the first sequence becomes 3 real tokens + 2 padding. For any query position in the first sequence, the padding mask row is $[0,0,0,-\infty,-\infty]$ — positions 3 and 4 are blocked regardless of what score they'd otherwise get. Without this mask, "cat" might assign, say, 15% of its attention weight to meaningless padding embeddings, diluting genuine signal and letting a real token's output depend partly on filler carrying no information at all.

A causal mask's allowed-position pattern is a lower-triangular matrix — think of this explorer's grid similarly: only entries on or below the diagonal survive, everything above is forced to zero contribution.

What this means in practice

Causal masking is what makes autoregressive generation (GPT-style decoder-only models) trainable in parallel across a whole sequence at once — the sequence is scored simultaneously, but the mask ensures the training signal at each position only reflects what that position could legitimately have seen at generation time. Padding masking is unavoidable in any batched training on variable-length sequences and interacts directly with The KV Cache and Incremental Decoding, since a wrongly-masked padding token during incremental generation can corrupt every later step's attention.

Practice

1. For a 5-token causal mask, write out the allowed-position pattern for query position 2 (0-indexed).
2. Explain why adding $-\infty$ before softmax is preferred over simply zeroing out the attention weights after softmax.
3. A batch mixes a 4-token real sequence padded to 6 with a genuine 6-token sequence. Describe the padding mask row for query position 0 in the first sequence.