Queries, Keys and Values

A plain Sequence-to-Sequence Encoder-Decoder Models model compresses an entire input sequence into one fixed-size context vector, and that bottleneck degrades badly on long sequences because everything has to be squeezed through one summary. The fix seems obvious in hindsight: instead of forcing the decoder to work from one compressed summary, let it look back at the original input sequence directly, at every step, and decide for itself which parts of the input are relevant right now. That "which parts are relevant" decision is exactly what attention computes, and queries, keys and values are the mechanism that makes the decision learnable rather than hand-coded.

The analogy: a librarian search

Imagine searching a library. You arrive with a query — what you're actually looking for, phrased in your own words ("books about mean reversion in pairs trading"). Every book on the shelf has a key — a short descriptor, like a catalog tag, that summarizes what that book is about, independent of what anyone happens to be searching for. You compare your query against every book's key and get a relevance score for each — high for a close match, low for an unrelated book. Then you don't walk away with the keys — you walk away with the actual value: the book's contents. Attention runs this exact search, numerically, at every position of a sequence, simultaneously: every position emits a query (what it's looking for), every position (including itself) offers a key (what it has to offer) and a value (what gets returned if selected), and the output is a blend of values weighted by how well each key matched the query.

The maths

Given a set of input vectors $x_1, \dots, x_n$ (each, say, a token or timestep's embedding), three learned weight matrices project each input into a query, key, and value:

$$q_i = x_i W_Q, \qquad k_i = x_i W_K, \qquad v_i = x_i W_V$$

In words: the same input vector $x_i$ produces three different roles by being multiplied by three different learned matrices — $W_Q$, $W_K$, $W_V$. These are what the network actually learns during training; nothing about "query" or "key" is hand-designed, only the mechanism that uses them is fixed.

For a given query $q_i$, its attention score against every key $k_j$ in the sequence is a scaled dot product:

$$\text{score}(q_i, k_j) = \frac{q_i \cdot k_j}{\sqrt{d_k}}$$

In words: measure how well query $i$ and key $j$ align via a dot product (large when the two vectors point in a similar direction, i.e. when they "match"), then divide by $\sqrt{d_k}$ — the square root of the key dimension — purely to keep the scores from growing too large as $d_k$ increases, which would otherwise push the next step's softmax into a regime with vanishing gradients.

These scores across all $j$ are turned into weights that sum to 1 via softmax, then used to blend the values:

$$\text{Attention}(q_i) = \sum_{j=1}^{n} \text{softmax}_j\!\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right) v_j$$

In words: normalize the scores into a probability-like distribution over all $n$ positions, then take a weighted average of every position's value vector using those weights. Position $i$'s output is a blend of everyone's values, leaning heavily on whichever positions its query matched best. Unlike a fixed-size context vector, this readout can pull directly from any position in the sequence, however far away, with a weight the model computes fresh for every query — nothing forces distant information to decay the way it does in a recurrent chain.

Worked example 1: computing attention weights by hand

Take $n = 3$ positions with 2-dimensional keys and a query $q = (1, 0)$. Keys: $k_1 = (1, 0)$, $k_2 = (0, 1)$, $k_3 = (0.7, 0.7)$. Use $d_k = 2$, so $\sqrt{d_k} \approx 1.414$.

Dot products: $q \cdot k_1 = 1$, $q \cdot k_2 = 0$, $q \cdot k_3 = 0.7$. Scaled scores:

$$\frac{1}{1.414} \approx 0.707, \qquad \frac{0}{1.414} = 0, \qquad \frac{0.7}{1.414} \approx 0.495$$

Softmax over $(0.707, 0, 0.495)$: exponentiate, $e^{0.707} \approx 2.028$, $e^{0} = 1$, $e^{0.495} \approx 1.640$, sum $\approx 4.668$.

$$\text{weights} \approx (0.434,\ 0.214,\ 0.351)$$

Position 1's key matched the query best (raw dot product 1, the highest), so it gets the largest weight, 0.434 — but position 3's key, only a partial match, still earns a meaningful 0.351, not zero. This is the whole point of softmax here: it's a soft, graded lookup, not a hard "pick the single best match and ignore everything else."

Worked example 2: blending values with those weights

Continue the example with values $v_1 = (2, 0)$, $v_2 = (0, 2)$, $v_3 = (1, 1)$. The output is the weighted sum using the weights just computed, $(0.434, 0.214, 0.351)$:

$$\text{out} = 0.434(2,0) + 0.214(0,2) + 0.351(1,1)$$ $$= (0.868, 0) + (0, 0.428) + (0.351, 0.351) = (1.219,\ 0.779)$$

The output vector $(1.219, 0.779)$ is not any single input's value — it's a genuine blend, dominated by $v_1$ (since it got the largest weight) but pulled toward $v_2$ and $v_3$ proportionally. Change the query and every weight recomputes fresh; there's no fixed routing, which is exactly what lets the same mechanism serve completely different lookups at every position, at every layer, without re-architecting anything.

$W_Q$, $W_K$, $W_V$ are each just a learned linear map applied to the input, the same kind of matrix transform this explorer visualizes as a stretch and rotation of a shape. Attention's expressive power comes from three differently-trained transforms of the same input feeding into a similarity comparison, not from any one of them individually.

What this means in practice

This query-key-value lookup is the mechanism underneath every transformer, and by extension underneath most modern large language models and increasingly, sequence models used for price and order-flow forecasting. Its practical draw over a recurrent architecture is that any position can attend directly to any other position in one step, with no decay over distance — useful when a relevant piece of context (a macro print, a large block trade) could be arbitrarily far back in the window and needs to be found on its merits, not on its recency. Its cost is that comparing every query against every key is $O(n^2)$ in sequence length, which is exactly the scaling problem that motivated the state-space alternatives in Mamba and Selective State Spaces.