Scaled Dot-Product Attention and the √d Factor

Transformer attention scores how relevant one token is to another using a dot product between a query vector and a key vector. That works well in low dimensions, but as the vectors get wider — modern models use dimensions in the hundreds — the raw dot products start growing large simply as a side effect of having more terms to sum, not because the tokens have become more related. Large scores fed into a softmax push it toward an almost one-hot output, and the training signal (gradients) through a nearly one-hot softmax collapses to almost nothing. The √d scaling factor is a small correction that exists to stop this from happening.

The analogy: averaging opinions from a growing committee

Imagine measuring how much two people agree by adding up how similarly they answer yes/no questions, scoring +1 for agreement and -1 for disagreement, then summing. With 4 questions, the max possible score is 4. With 400 questions, the max is 400 — not because people agree "400 times more," purely because more terms are summed. Feeding that raw sum into a rule like "higher score means exponentially more attention" would make a 400-question committee look wildly, artificially more extreme than a 4-question one, purely from arithmetic. The fix is obvious: normalize by the number of questions. Scaled dot-product attention does the equivalent normalization for dot products over $d$ dimensions.

The mechanics: why dot products scale with √d, and the fix

For a query $q$ and key $k$, each with $d$ independent components roughly of unit variance, the dot product is a sum of $d$ terms:

$$q \cdot k = \sum_{i=1}^{d} q_i k_i$$

If each term $q_i k_i$ has mean 0 and variance roughly 1 (a standard modeling assumption for randomly-initialized components), then the variance of the sum of $d$ roughly-independent terms is $d$ times the variance of one term, so the standard deviation of the dot product grows like $\sqrt{d}$. In words: the typical size of the raw dot product grows with the square root of the dimension, purely from summing more terms — the same arithmetic effect as the committee example, just phrased in the language of variance. Scaled dot-product attention divides the raw score by exactly $\sqrt{d}$ before applying softmax:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right) V$$

In words: compute every query-key dot product, shrink each one by dividing by the square root of the key dimension to undo the dimension-driven growth, turn the resulting scores into weights with softmax, then use those weights to blend the value vectors — the same score-softmax-blend pattern as Bahdanau and Luong Attention, with one extra division inserted specifically to keep the scores in a numerically healthy range regardless of $d$.

Worked example 1: comparing softmax with and without scaling

Say two of three raw dot-product scores are $[8, 2, 0]$, with $d = 64$, so $\sqrt{d} = 8$. Unscaled softmax: $\exp(8)=2981$, $\exp(2)=7.4$, $\exp(0)=1$, weights $\approx [0.997, 0.0025, 0.0003]$ — essentially all the weight on one position, with the other two contributing almost nothing and receiving almost no gradient during training. Scaled by $\sqrt{64}=8$: scores become $[1.0, 0.25, 0]$. Softmax: $\exp(1.0)=2.72$, $\exp(0.25)=1.28$, $\exp(0)=1$, weights $\approx [0.53, 0.25, 0.22]$ — a much softer, more gradient-friendly distribution that still favors position 1 but doesn't crush the others to zero.

Worked example 2: how the effect grows with dimension

Suppose typical dot product magnitude scales as $\sqrt{d}$. At $d=64$, a "typical large" score might be around $\sqrt{64} \times 3 = 24$. At $d=512$ (a wider model), the same relative signal produces a score around $\sqrt{512} \times 3 \approx 68$ — nearly 3x larger, purely from the dimension increasing 8x. Dividing by $\sqrt{d}$ in each case brings both back down to the same "typical large" scaled score of about 3, restoring comparable softmax behavior regardless of model width.

Raw dot-product magnitude grows like a square-root curve as dimension increases — dividing by $\sqrt{d}$ is precisely the correction that flattens that growth back to a constant scale, whatever $d$ happens to be.

What this means in practice

Every standard transformer, from the original architecture through today's large language models, applies this exact $\sqrt{d}$ division inside every attention head — it's not an optional tuning knob, it's load-bearing for training stability at the widths modern models use. Getting it wrong (or omitting it) when implementing attention from scratch is a classic silent bug: training doesn't crash, it just trains badly, with gradients vanishing through a saturated softmax in a way that's easy to misdiagnose as a learning-rate or initialization problem.

Practice

1. For raw scores $[4, 4, 0]$ and $d=16$ (so $\sqrt{d}=4$), compute the softmax weights with and without scaling and compare how peaked each is.
2. Explain in one sentence why the scaling factor is $\sqrt{d}$ specifically, rather than $d$ or a constant.
3. Why would omitting √d scaling be a "silent" bug rather than one that immediately crashes training?