The Transformer Architecture

Recurrent models pass information along a chain: step 1 tells step 2, which tells step 3. Two problems follow. Information from far back has to survive every intermediate hop, and every step must wait for the one before it, so a thousand-step sequence takes a thousand sequential operations no matter how many GPUs you own. The transformer removes the chain. Every position reads every other position directly, in parallel, and the model decides for itself which ones are worth reading.

The analogy: everyone asks the room

Imagine a trading floor where each day of price history is a person holding a note. To form its view, each person stands up and announces what it is looking for — "I need days where volume spiked" — and everyone else holds up a tag saying what they have — "big volume day", "quiet drift day". Each listener compares its request to every tag, decides how much attention to give each person, and takes a weighted blend of their notes.

Those three roles are the whole mechanism, and they have names:

• the query $q$: what this position is looking for,
• the key $k$: what that position advertises about itself,
• the value $v$: the content it hands over if you attend to it.

All three are just small learned linear projections of the same input. Nothing mystical happens; the model learns three different ways of looking at each token.

The formula

For a whole sequence, stack the queries into a matrix $Q$, keys into $K$, values into $V$:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V .$$

Read it right to left in four moves. $QK^{\top}$ takes the dot product of every query with every key, giving a score for each pair. Dividing by $\sqrt{d_k}$ — the square root of the key dimension — keeps those scores from growing simply because the vectors are long. The softmax turns each row of scores into positive weights that add to one. Multiplying by $V$ averages the values using those weights.

Worked example: attention by hand

Three days of history, keys of dimension $d_k = 2$. Today (day 3) issues the query $q = (1.0,\ 0.5)$. The three keys and their scalar values are

$$k_1 = (0.2,\ 1.0),\ v_1 = 2.0; \qquad k_2 = (1.2,\ 0.4),\ v_2 = -1.0; \qquad k_3 = (0.9,\ 0.6),\ v_3 = 0.5 .$$

Step 1, score every pair. Dot products:

$$q \cdot k_1 = 0.2 + 0.5 = 0.7, \qquad q \cdot k_2 = 1.2 + 0.2 = 1.4, \qquad q \cdot k_3 = 0.9 + 0.3 = 1.2 .$$

Step 2, scale. Divide each by $\sqrt{2} = 1.414$: the scores become $0.495$, $0.990$, $0.849$.

Step 3, softmax. Exponentiate: $e^{0.495} = 1.641$, $e^{0.990} = 2.691$, $e^{0.849} = 2.337$. They sum to $6.669$, so the weights are

$$0.246, \qquad 0.404, \qquad 0.350 .$$

Step 4, average the values.

$$0.246 (2.0) + 0.404 (-1.0) + 0.350 (0.5) = 0.492 - 0.404 + 0.175 = 0.263 .$$

Today's output is $0.263$. It leaned most on day 2 because day 2's key matched today's query best — and it worked that out from the data, not from a fixed rule about how far back to look.

Worked example: why divide by the square root

Scores are dot products of $d_k$ terms, so they grow roughly like $\sqrt{d_k}$ as the dimension rises. Take the same three days but with $d_k = 64$ instead of 2. The raw dot products come out about $\sqrt{32} = 5.66$ times larger: $3.96$, $7.92$, $6.79$.

Softmax those unscaled and the weights become

$$0.014, \qquad 0.745, \qquad 0.241 .$$

Day 2 now soaks up three quarters of the attention and day 1 has effectively vanished — not because the relationships changed, but because the vectors got longer. Worse, a saturated softmax has almost no gradient, so training stalls.

Now divide by $\sqrt{64} = 8$: the scores return to $0.495$, $0.990$, $0.849$ and the weights to $0.246$, $0.404$, $0.350$ — exactly the two-dimensional answer. The scaling makes attention behave the same at any width.

The softmax is exponential, which is what makes it so easy to saturate. Steepen the exponential curve below and watch how a modest gap in the input turns into a huge gap in the output — that is the effect the $\sqrt{d_k}$ divisor is holding in check.

The rest of the block

Attention alone is a weighted average, which is a fairly weak operation. Four pieces surround it:

• Multi-head. Run several attentions in parallel with separate projections, then concatenate. One head can track momentum, another liquidity, another the earnings date. Splitting a 512-wide model into 8 heads of 64 costs nothing extra in compute.
• Positional encoding. Attention is order-blind — shuffle the days and the maths is identical. A position signal is added to each input so the model can tell "yesterday" from "a year ago".
• Residual connections and normalisation. Each sublayer computes $x + \text{sublayer}(x)$ followed by a normalisation, which is what lets dozens of blocks stack without the signal degrading.
• A position-wise feed-forward layer. A small two-layer network applied to each position separately, giving the model the nonlinearity that averaging cannot provide.

Stack that block $N$ times and you have a transformer.

What this means in practice

Transformers arrived in finance on the promise of modelling long-range structure in returns, order flow and text. What actually holds up:

• Cost is quadratic in sequence length. A thousand ticks means a million scores per head per layer. Doubling the window quadruples the bill. This is the reason people bucket ticks into bars before feeding a transformer.
• They are data-hungry. With no recurrence prior, a transformer must learn the very notion of "adjacent in time" from scratch. Twenty years of daily bars is roughly 5,000 rows — far too little. They earn their keep on tick data, cross-sectional panels of thousands of names, and text.
• Causal masking is not optional. In language modelling the mask is a nicety; in trading it is the difference between a result and a fantasy. Any position attending to a later timestamp has leaked the future.

The transformer's real gift is that path length between any two positions is one, not $n$. That is why it learns long-range structure where gated recurrent cells struggle — and why the bill arrives as memory rather than as forgetting.