Attention Is Not Explanation

A transformer model reading news headlines to score sentiment gives one word in a headline a much higher attention weight than the rest. It is natural to conclude the model "focused on" that word and that this is why it made the prediction it did. This intuition is widespread and it is only sometimes true. Attention weights are a real, computed part of the model's forward pass — they are not made up — but treating them as an explanation of the model's reasoning is a stronger claim than the mechanism actually supports.

What attention weights actually are

In an attention layer, each output position computes a weighted average of value vectors from every input position, and the weights (which sum to one across the inputs) are what people call "attention." A high weight on a token means that token's value vector contributed heavily to the weighted sum at that step of computation — see Multi-Head Attention for the mechanics. That is a precise, verifiable fact about the arithmetic. "This token mattered a lot to the model's final decision" is a different, much larger claim, because the final decision depends on everything downstream of that weighted sum too — subsequent layers, other attention heads, residual connections — any of which could amplify, dampen, or entirely override what one attention weight suggests.

Two ways the link breaks

Weights don't determine outputs uniquely. Research testing this directly has found that you can often substitute a very different attention distribution over the same inputs — even a close-to-uniform one — retrain just the downstream layers to compensate, and get nearly the same prediction out. If multiple, quite different attention patterns are compatible with the same output, no single one of them can be the explanation of that output.

High attention doesn't imply high impact. Removing or zeroing out the token a model attended to most heavily sometimes barely changes the prediction, while removing a token that received low attention sometimes flips it — the causal test that attribution methods like Shapley values are explicitly built around, and which raw attention weights make no attempt to satisfy.

effect of removing word shares plunge despite strong earnings beat

Worked example

Score the headline "shares plunge despite strong earnings beat" for sentiment. Suppose the model attends most heavily to "plunge" (weight 0.31) and "earnings" (weight 0.24), and least to "despite" (weight 0.04) — attention alone would suggest "plunge" is driving the negative call. Now test causally: delete "despite" and re-run the model on "shares plunge strong earnings beat." The prediction flips from negative to positive, because "despite" is the word that inverts the relationship between the plunge and the earnings beat — grammatically small, causally decisive. Deleting "plunge" itself, by contrast, only shifts the score partway. The word with the least attention had the largest measured effect on the output; the word with the most attention did not.

Where quants meet this

Transformer-based models are increasingly used on news, filings and transcripts to generate trading signals (see The Transformer Architecture), and attention visualisations are a common way vendors and internal teams present "why the model said what it said." Before trusting one for model risk sign-off or feature debugging, run the causal check directly: perturb or remove the highly-attended tokens and confirm the output actually moves, the same discipline Counterfactual Explanations applies more generally.

The response, and where it landed

The finding that attention weights can be swapped out with little change in output was itself contested — a widely cited follow-up paper argued that under a fairer test (comparing attention distributions that are actually plausible for the model to have learned, rather than adversarially constructed alternatives), attention weights do carry meaningful, if imperfect, information about model behaviour in many settings. The honest current position is neither "attention is a rigorous explanation" nor "attention is meaningless," but somewhere in between: attention weights are a useful, cheap, informal signal worth looking at, and a poor substitute for an explanation with actual guarantees behind it, such as a Shapley-based attribution or a direct perturbation test.

What to use instead when it matters

When a decision genuinely needs a defensible account of what drove it — a model-risk sign-off, a client-facing "why did this signal fire" report — the fix is not to distrust attention and stop there, but to layer a method with an actual causal or axiomatic guarantee on top: perturbation-based testing (remove or mask a token and measure the output change directly, as in the headline example), or a formal attribution method like Shapley values, applied to the transformer's full output rather than read off one internal layer's weights. Attention maps remain a fine first pass for building intuition about a model quickly — the mistake is treating that first pass as the final word.