Multi-Head Attention

A single query-key-value attention computation, as described in Queries, Keys and Values, produces exactly one weighting per position — one answer to "what should I attend to." But relevance isn't one-dimensional. In a price series, one useful notion of "what's relevant to this bar" might be "the most recent bar with a similar volume spike"; a completely different, equally valid notion might be "the bar exactly one trading week ago, for seasonality"; a third might be "whichever bar started the current trend." A single attention head has to compress all of these into one weighting scheme, using one shared $W_Q$, $W_K$, $W_V$. Multi-head attention's fix is not to make one head cleverer, but to run several smaller heads side by side, each with its own independently learned projections, so each head is free to specialize in a different kind of relationship.

The analogy: a panel of specialist reviewers

A publisher sends a manuscript to several reviewers instead of one, and deliberately picks reviewers with different specialties — one checks the science, one checks the prose, one checks the market fit. Each reviewer reads the same manuscript but pays attention to different things and produces a different assessment; none of them individually sees the whole picture, but the editor combines all their reports into one final verdict that's richer than any single review could have been. Multi-head attention runs several "reviewers" — heads — over the same input sequence in parallel. Each head has its own learned $W_Q$, $W_K$, $W_V$, so each head can, over the course of training, specialize in noticing a different kind of pattern, and their outputs are combined at the end into one richer representation.

The maths

Instead of one set of projections operating on the full dimensionality $d$, split into $H$ heads, each operating on a smaller dimension $d_k = d / H$. Each head $h$ computes its own attention output independently, exactly as in Queries, Keys and Values but with its own weight matrices:

$$\text{head}_h = \text{Attention}(X W_Q^{(h)},\ X W_K^{(h)},\ X W_V^{(h)})$$

In words: run the exact same attention computation $H$ separate times on the same input $X$, but with $H$ different, independently trained sets of query/key/value projection matrices, each smaller than a single full-size projection would be. Head $h$'s $W_Q^{(h)}, W_K^{(h)}, W_V^{(h)}$ share no weights with head $h'$'s — during training, nothing stops them from converging to similar behavior, but nothing forces them to either, and in practice they tend to specialize.

The $H$ heads' outputs, each a vector of dimension $d_k$, are concatenated back into one vector of the original dimension $d$, then passed through one more learned linear layer to mix them:

$$\text{MultiHead}(X) = \text{concat}(\text{head}_1, \dots, \text{head}_H)\, W_O$$

In words: stack all $H$ heads' outputs side by side into one long vector, then apply a final learned matrix $W_O$ that lets the model combine information across heads — so the output at each position can draw on whatever mixture of "similar volume spike," "same time last week," and "trend origin" that particular position needs, rather than being locked into just one head's finding.

Splitting into $H$ heads of size $d_k = d/H$ rather than running $H$ full-size heads keeps the total compute roughly the same as one big head — you're trading one wide lookup for several narrower, independently-specialized ones at comparable total cost, not multiplying the cost by $H$.

Worked example 1: two heads finding different things on the same input

Take a toy sequence of 3 positions with a query at position 3 (a "signal" bar). Suppose head 1's learned projections make it good at detecting magnitude similarity and its raw dot-product scores against the three keys come out $(0.2, 0.1, 3.0)$ — strongly favoring itself (position 3), meaning "attend mostly to your own recent value." Head 2's projections make it sensitive to something else entirely, say positional distance for a weekly pattern, and its scores come out $(2.5, 0.3, 0.2)$ — favoring position 1.

Running softmax on each independently: head 1's weights are close to $(0.06, 0.05, 0.89)$ — it attends almost entirely to position 3 itself. Head 2's weights come out close to $(0.85, 0.08, 0.07)$ — it attends almost entirely to position 1. Concatenating head 1's output (dominated by position 3's value) with head 2's output (dominated by position 1's value) into one vector, then passing through $W_O$, produces a final representation for position 3 that carries information from both "what's happening right now" and "what happened at the relevant earlier point" — something neither head alone could represent, and something a single-head attention layer, forced to pick one weighting, would have had to compromise between.

Worked example 2: counting the dimension split

Suppose the model dimension is $d = 512$ and it uses $H = 8$ heads. Each head operates on:

$$d_k = \frac{512}{8} = 64$$

Each head's $W_Q^{(h)}$ is a $512 \times 64$ matrix (512 input dimensions in, 64 head-dimensions out), and likewise for $W_K^{(h)}$, $W_V^{(h)}$. Parameters per head, per projection: $512 \times 64 = 32{,}768$. Three projections per head: $98{,}304$. Across 8 heads: $8 \times 98{,}304 = 786{,}432$. Compare to a single full-size head with $d_k = 512$ (no splitting): $3 \times (512 \times 512) = 786{,}432$ — identical total parameter count. Splitting into 8 narrower heads costs nothing extra in parameters or compute versus one wide head; what's bought for free is the ability for each of the 8 sub-spaces to specialize independently, which is why multi-head attention is essentially always preferred over single-head at the same total width.

Each head's $W_Q$, $W_K$, $W_V$ is an independent linear transform of the same input, the way this explorer's matrix stretches and rotates the same unit circle differently depending on which matrix is applied — different heads are, in effect, different transforms of the identical starting point, each surfacing a different structure.

What this means in practice

In a transformer applied to financial sequences, different heads empirically tend to specialize — some attending to very recent bars (short-range momentum-like patterns), others to fixed periodic offsets (weekly or monthly seasonality), others to a small number of unusually large moves regardless of how far back they occurred (regime-defining events). This is not guaranteed or interpretable in every case, but it is the mechanism that gives a transformer more representational flexibility than a single-head version at the same parameter budget, and it's a large part of why transformers generalize better than plain RNNs on tasks with multiple, simultaneously-relevant kinds of dependency.