Bahdanau and Luong Attention

An early encoder-decoder model reads an entire input sequence and squeezes everything it learned into one fixed-size vector, then asks the decoder to generate the whole output from that single vector alone. For a five-word sentence that works fine. For a fifty-word sentence, it does not: cramming fifty words of meaning into one vector of fixed size loses detail, the same way summarizing a whole novel in one sentence loses detail no matter how good the summary is. The decoder needs a way to look back at the original input, position by position, not just its one compressed summary.

The analogy: translating with the source text open on the desk

Imagine translating a long paragraph from memory versus translating it with the original text open in front of you, so you can glance back at whichever part is relevant to the word you're writing right now. Memorizing the whole paragraph as one mental summary (the old fixed-vector approach) works for a sentence but falls apart for a paragraph. Keeping the source open and attending to the relevant phrase for each word you write — sometimes the beginning, sometimes the middle — is what attention gives a decoder: instead of one static summary, a fresh, re-weighted look back at the full input for every single output step.

The mechanics: scoring, weighting, and blending

At each decoder step $t$, attention computes a score between the decoder's current state $s_t$ and every encoder hidden state $h_i$ (one per input position $i$), turns those scores into weights that sum to 1, and blends the encoder states using those weights into a context vector $c_t$:

$$e_{t,i} = \text{score}(s_t, h_i), \qquad \alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_j \exp(e_{t,j})}, \qquad c_t = \sum_i \alpha_{t,i}\, h_i$$

In words: score how relevant each input position is to what the decoder is producing right now, turn those relevance scores into a probability distribution over positions (the softmax), then take a weighted average of the encoder's representations using that distribution as the weights. The context vector $c_t$ changes at every decoder step — it is a fresh, targeted look at the input, not one fixed summary reused throughout.

Bahdanau's original scoring function is additive: a small neural network combines $s_t$ and $h_i$, $e_{t,i} = v^\top \tanh(W_1 s_t + W_2 h_i)$, computed before the decoder's current step updates. Luong's variant is multiplicative (dot-product-based), $e_{t,i} = s_t^\top W h_i$ or simply $s_t^\top h_i$, computed after the decoder state updates, and is cheaper to compute since it's just a matrix multiply rather than a small trained network — this multiplicative form is the direct ancestor of the scaled dot-product attention used in transformers.

Worked example 1: computing attention weights by hand

Say a decoder state $s_t$ has a dot product with three encoder states of $e = [2.0, 0.5, -1.0]$ (Luong-style). Softmax: $\exp(2.0)=7.39$, $\exp(0.5)=1.65$, $\exp(-1.0)=0.37$, sum $= 9.41$, giving $\alpha = [0.785, 0.175, 0.039]$. The context vector is $0.785\, h_1 + 0.175\, h_2 + 0.039\, h_3$ — almost entirely position 1, a little of position 2, position 3 essentially ignored. That is attention concretely: a soft, differentiable "mostly look here" pointer.

Worked example 2: attention shifting across output steps

Translating "the red car" into another language with the words reordered as "car red the," a well-trained attention mechanism would produce weights like $\alpha_1 = [0.05, 0.10, 0.85]$ (mostly "car") when generating the first output word, then $\alpha_2 = [0.10, 0.80, 0.10]$ (mostly "red") for the second, then $\alpha_3 = [0.85, 0.10, 0.05]$ (mostly "the") for the third. The weight distribution moves across the input as the decoder works through the output — exactly the "glancing back at the relevant phrase" behavior from the analogy, discovered automatically from data with no hand-built alignment rules.

Softmax turns a set of raw scores into a distribution that sums to 1, just like this curve's area always integrates to 1 — attention weights are exactly this kind of normalized distribution, one such distribution freshly computed at every decoder step.

What this means in practice

Bahdanau and Luong attention fixed the fixed-vector bottleneck in encoder-decoder models years before transformers existed, and the mechanism itself — score, softmax, weighted sum — survives almost unchanged inside every transformer's Scaled Dot-Product Attention and the √d Factor. The distinction between attending over a different sequence (the encoder's output, as here) versus attending within the same sequence is the difference between this cross-attention pattern and self-attention, formalized later as Cross-Attention Between Encoder and Decoder.

Practice

1. Compute softmax weights for raw scores $e = [1.0, 1.0, 1.0]$ and explain what the resulting context vector represents.
2. State the practical computational difference between Bahdanau's additive scoring and Luong's multiplicative scoring.
3. Explain why a fixed-size context vector becomes a worse bottleneck as sequence length grows.