Encoder-Only, Decoder-Only and Encoder-Decoder Designs

Say you want a model to read a research report and tell you whether it is bullish or bearish. Then say you want a different model to write next quarter's report from scratch, one word at a time. Both jobs use the same building block — the attention layer — but they need opposite rules about who is allowed to look at whom. The first model should let every word see every other word, including ones later in the document, because the whole document is sitting there to be read. The second model must never let a word see what comes after it, because at generation time there is no "after" yet — the model is inventing it. That single rule, about which tokens are visible to which, is the entire difference between the three transformer families.

The analogy: an exam room vs a live commentator

Picture a student sitting an open-book exam on a passage they've already been handed in full. They can flip back and forth, cross-reference the third paragraph against the first, and answer any question about the whole text at once. That's an encoder: it sees the entire input simultaneously and builds a rich understanding of it. No word is hidden from any other word.

Now picture a sports commentator narrating a match live. They can reference everything that has already happened — the goal five minutes ago, the substitution at half-time — but they cannot know or describe what happens in the next minute before it happens. Each word they say is produced only from the words said so far. That's a decoder: strictly one-directional, always causal, generating the next token from everything before it and nothing after.

An encoder-decoder is both jobs stapled together: a student who reads the whole exam passage first (encoder), then writes an essay about it one word at a time while still being allowed to glance back at the passage (decoder, with an extra channel into the encoder's understanding). This is what translation needs — read the whole French sentence, then generate the English one word by word, checking back against the French at every step.

The mechanism: masking, in one formula

Every attention layer computes, for each query token, a weighted average of value vectors from other tokens, where the weights come from how well the query matches each token's key. Call the raw compatibility score between query $i$ and key $j$ as $s_{ij}$, a single number measuring how relevant token $j$ is to token $i$. Before turning these scores into weights with softmax, a mask $M$ is added:

$$\text{weight}_{ij} = \text{softmax}_j\left(s_{ij} + M_{ij}\right), \qquad M_{ij} = \begin{cases} 0 & \text{token } j \text{ is visible to token } i \\ -\infty & \text{token } j \text{ is hidden from token } i \end{cases}$$

In words: the mask is a lookup table of "allowed" and "forbidden" pairs. Adding zero leaves a score untouched; adding negative infinity sends that pair's weight to exactly zero after softmax, because $e^{-\infty} = 0$. Nothing about the attention math changes across the three architectures — only which entries of $M$ are $-\infty$.

• Encoder-only (BERT-style): $M_{ij} = 0$ everywhere. Full visibility, both directions.
• Decoder-only (GPT-style): $M_{ij} = -\infty$ whenever $j > i$ — a token cannot see anything to its right. This is called a causal mask.
• Encoder-decoder (T5, original Transformer): the encoder stack uses a full mask on the input; the decoder stack uses a causal mask on the output and an unmasked cross-attention layer that lets every decoder token see every encoder token.

Worked example 1: masking a 4-token sentence

Take the sentence "rates rose again today" as four tokens, positions 1 through 4. Suppose the raw compatibility scores from token 3 ("again") to all four tokens are $s = [1.0,\ 2.0,\ 3.0,\ 0.5]$ (token 3 attending to itself most, position 4 least so far).

Encoder-only: no masking, so all four scores go straight into softmax: $e^{1.0}=2.72$, $e^{2.0}=7.39$, $e^{3.0}=20.09$, $e^{0.5}=1.65$, summing to $31.85$. Weights are roughly $0.085,\ 0.232,\ 0.631,\ 0.052$ — token 3 mostly attends to token 3 and a bit to token 2, but token 4 ("today") still gets a nonzero say even though it comes later in the sentence.

Decoder-only: token 3 may only see tokens 1–3, so $M_{3,4}=-\infty$ and the fourth score is discarded before softmax. Renormalizing over just $[1.0, 2.0, 3.0]$: sum $= 2.72+7.39+20.09=30.20$, weights $\approx 0.090,\ 0.245,\ 0.665$. Token 4 gets weight exactly zero — it has no influence at all, because at the moment token 3 is being processed during generation, token 4 doesn't exist yet.

Worked example 2: why the choice matches the job

A sentiment classifier reading "the trade did not work out" needs "not" to modify "work out" regardless of which word appears first when the model attends — bidirectional context lets "work" attend backward to "not" and forward to "out" simultaneously, both in a single encoder pass. An encoder-only model with a classification head on top does this in one shot.

A price-generating model producing "the fund posted a [BLANK]" one token at a time cannot use a bidirectional encoder, because at generation time there genuinely is no future token to look at — "gain" or "loss" hasn't been decided yet. It must be decoder-only (or the decoder half of an encoder-decoder), predicting token $t{+}1$ from tokens $1 \ldots t$ only, then feeding its own output back in as the next input. This autoregressive loop — predict one token, append it, repeat — is why decoder-only generation is sequential and, without a KV cache, expensive.

What this means in practice

Most large language models deployed today — the ones you chat with, the ones drafting research summaries — are decoder-only, because a single causal model can be prompted to do classification, generation, and translation all at once just by phrasing the task as "predict the next token," and it scales the most cleanly to huge sizes. Encoder-only models remain the workhorses for embeddings, retrieval and classification, where you want one fixed, richly contextual representation of a whole passage rather than a stream of generated text. Encoder-decoder models persist wherever the task is genuinely "transform sequence A into sequence B" — machine translation, summarization with a strict source document, or structured extraction from a filing.

Practice

1. A model needs to read an entire earnings call transcript and output a single risk score. Which architecture fits, and why would adding a causal mask hurt it?
2. Sketch the mask $M$ for a 3-token decoder as a 3×3 grid of 0 and $-\infty$ entries.
3. In an encoder-decoder translating "rates rose" into French, does the decoder's cross-attention step need a causal mask against the encoder's tokens? Explain using the visibility rule above.