The Self-Attention Mechanism

An RNN reading a long report has to compress everything it has read so far into one running summary, and by page 40 details from page 2 have often faded or been overwritten. Self-attention takes a completely different approach: instead of a relay of running summaries, every word gets to look back — directly, in one step — at every other word in the document and decide for itself which ones matter most for understanding it right now. No compression bottleneck, no forgetting through a long chain of updates.

Think of a research analyst reading a earnings call transcript. When they hit the sentence "it grew significantly," they don't recall a compressed summary of everything said before — they scan back and directly notice which earlier sentence "it" refers to ("cloud revenue," three sentences up) and weight that connection heavily while ignoring most of the rest. Self-attention formalizes exactly that: for each word, compute a relevance score against every other word, turn those scores into weights, and build a new representation as a weighted blend.

The mechanics, one symbol at a time

Every input position (a word, a time step, a bar of price data) is turned into three vectors by three learned weight matrices: a query $q$ (what this position is looking for), a key $k$ (what this position offers to others, as a label to be matched against), and a value $v$ (the actual content this position contributes if chosen). For position $i$, its new representation is:

$$\text{Attention}(q_i, K, V) = \sum_j \text{softmax}\left(\frac{q_i \cdot k_j}{\sqrt{d}}\right) v_j$$

In words: for query $i$, compare it against every key $k_j$ using a dot product (a similarity score — high when the two vectors point in a similar direction), scale by $\sqrt{d}$ (a fixed constant that keeps the scores from growing too large as the vector dimension $d$ grows, which would otherwise make the next step too extreme), then pass all the scores for position $i$ through softmax — a function that turns any list of numbers into positive weights summing to 1, favoring the largest inputs. Those weights say how much position $i$ should attend to each position $j$. The output is a weighted average of every position's value vector, weighted by how relevant each one is to query $i$.

Worked example 1: attention scores for one query, three positions

Three tokens with 2-dimensional keys and query already computed (numbers simplified for hand computation). Query for token "it": $q = (1, 0)$. Keys: $k_1 = (1, 0)$ ("cloud revenue"), $k_2 = (0, 1)$ ("last year"), $k_3 = (0.8, 0.2)$ ("grew").

Dot products: $q \cdot k_1 = 1(1)+0(0) = 1$. $q \cdot k_2 = 1(0)+0(1) = 0$. $q \cdot k_3 = 1(0.8)+0(0.2) = 0.8$.

Scale by $\sqrt{d} = \sqrt{2} \approx 1.41$: scores are $(0.71, 0, 0.57)$.

Softmax: exponentiate, $e^{0.71}\approx 2.03$, $e^{0}=1$, $e^{0.57}\approx 1.77$; sum $\approx 4.80$. Weights: $(0.42, 0.21, 0.37)$.

"It" ends up attending most heavily (42%) to "cloud revenue," some (37%) to "grew," and least (21%) to "last year" — a sensible allocation for resolving what "it" refers to, produced entirely by dot-product similarity, with no hand-coded grammar rule.

Worked example 2: the output as a weighted blend

Using the same weights $(0.42, 0.21, 0.37)$ and 1-dimensional values for simplicity: $v_1 = 5.0$ ("cloud revenue," a strong, specific concept), $v_2 = 1.0$ ("last year," a weak time marker), $v_3 = 3.0$ ("grew," a moderate action concept).

$$\text{output} = 0.42(5.0) + 0.21(1.0) + 0.37(3.0) = 2.10 + 0.21 + 1.11 = 3.42$$

"It"'s new representation, $3.42$, is now a blend dominated by "cloud revenue" and "grew," with "last year" barely contributing — the model has effectively resolved the reference by mixing in the content of the words it matters most, in a single computation involving every position at once, not a step-by-step relay.

The query-key dot product is, geometrically, exactly the operation this explorer visualizes one level down: vectors are compared by how aligned they are after being placed in a learned space, and that alignment — not raw word order — is what attention scores measure.

What this means in practice

Self-attention is the core operation inside transformers, which now underlie most state-of-the-art sequence models — language models, and increasingly time-series and order-book models in finance. Its two big practical advantages over recurrence: every position can be processed in parallel (no waiting for step $t-1$ before computing step $t$), and the path length between any two positions is exactly one step, regardless of how far apart they are, which avoids the vanishing-gradient-over-many-steps problem recurrent networks face. The cost is quadratic: computing attention scores between every pair of $T$ positions takes $O(T^2)$ work and memory, which is why very long sequences (say, years of daily bars, or full limit-order-book histories) need specialized, sparse, or windowed variants of attention to stay tractable.