Cross-Attention Between Encoder and Decoder

Self-attention, the mechanism inside most transformer blocks, lets every position in a sequence attend to every other position within the same sequence. That's exactly right for understanding one passage of text on its own. But some tasks fundamentally involve two sequences — translating a sentence from one language to another, summarizing a document into a shorter passage, transcribing audio into text — where the output being generated needs to repeatedly refer back to a different sequence, the input, while it's being produced. Self-attention alone has no mechanism for that; you need attention between two distinct sequences, not within one.

The analogy: writing a summary with the source document open beside you

Self-attention is like proofreading a paragraph by re-reading itself, checking how its own sentences relate to each other. Cross-attention is like writing a summary of a completely different, longer document: as you write each sentence, you keep glancing back at the source, checking which part is relevant to the sentence you're currently drafting. The summary in progress is one sequence; the source document is a separate, fixed sequence you keep referring back to — never confusing "what I've written" with "what the source says," but constantly relating the two.

The mechanics: queries from one sequence, keys and values from another

Cross-attention uses the same score-softmax-blend attention formula as self-attention, but with a crucial difference in where each of the three inputs — queries, keys, and values — comes from:

$$\text{CrossAttn}(Q_{\text{dec}}, K_{\text{enc}}, V_{\text{enc}}) = \text{softmax}\!\left(\frac{Q_{\text{dec}} K_{\text{enc}}^\top}{\sqrt{d}}\right) V_{\text{enc}}$$

In words: the queries come from the decoder's current hidden states — "what am I trying to produce right now" — while the keys and values come from the encoder's finished output — "what does the source contain, and what should be retrieved." Self-attention, by contrast, draws $Q$, $K$, $V$ all from the same sequence. That's the single structural difference — the direct architectural descendant of Bahdanau and Luong Attention, generalized into the query/key/value framework.

A decoder block therefore contains three sub-layers: masked self-attention over its own partial output (so it doesn't peek ahead), cross-attention over the encoder's output (so it can consult the source), and a feedforward layer — versus an encoder block, which only needs self-attention.

Worked example 1: translation, step by step

Translating "the cat sat," three words in, the decoder is producing its third output word, built from what it has generated so far ("le chat…", say). Cross-attention computes this query against keys built from all three encoder positions ("the," "cat," "sat"), suppose yielding scores $[0.3, 0.2, 3.1]$ after scaling. Softmax gives weights roughly $[0.05, 0.04, 0.91]$ — the decoder correctly looks almost entirely at "sat" while producing the verb, even though "sat" is the last encoder position. Cross-attention has no notion of matching sequence positions; it retrieves whichever encoder position is relevant, regardless of where it sits in the source order.

Worked example 2: multiple cross-attention layers refining the same query

A 6-layer decoder performs cross-attention at every layer, not just once. Suppose at layer 1 the weights for a given decoder position are diffuse, $[0.3, 0.3, 0.4]$ — the model hasn't yet resolved what's relevant. By layer 6, after the decoder's representation is refined through several self-attention and feedforward steps, the same computation might sharpen to $[0.05, 0.05, 0.90]$. The decoder query gets progressively more informative through layers, so later cross-attention calls retrieve more precisely — the encoder's keys and values stay fixed throughout, but what the decoder asks for gets sharper.

Think of the encoder's positions as fixed points and the decoder's evolving queries as sweeping across them: cross-attention measures, at every decoder step, which encoder points the query currently lines up with most strongly — a relationship that sharpens layer by layer, much like a fit line summarizing a scatter better as information accumulates.

What this means in practice

Cross-attention is the architectural feature that defines an encoder-decoder transformer as distinct from an encoder-only (BERT-style) or decoder-only (GPT-style) model, discussed further in Encoder-Only, Decoder-Only and Encoder-Decoder Designs — a decoder-only model has no separate source to cross-attend to, so it uses only masked self-attention. It's essential wherever generation is conditioned on a distinct input: translation, summarization, image captioning, speech-to-text.

Practice

1. State which of Q, K, V come from the decoder and which from the encoder in cross-attention.
2. Explain why a decoder-only (GPT-style) model has no need for cross-attention.
3. In the translation example, why can the decoder attend most strongly to the last encoder position while only on its third output word?