Induction Heads and In-Context Learning

Show a language model "...Alice went to the market, Alice bought" and it will predict "apples" or some plausible continuation — but show it "...Zorblax went to the market, Zorblax bought" and it still completes the pattern sensibly, even though "Zorblax" never appeared in training. Nothing about that specific token was learned in advance; the model figured out, purely from the prompt, that whatever followed "Zorblax" the first time is likely to follow it the second time. This is in-context learning: adapting behaviour to new information within a single forward pass, no gradient update at all. A concrete, well-studied circuit inside transformers does exactly this — an induction head.

The analogy: finishing someone else's sentence

If a friend says "the meeting got moved to Thurs—" and pauses, you finish "—day" because you've heard that general pattern many times before. But if they say "the code is banana-forty-—" and pause, no general pattern helps — instead you scan back through the conversation for the last time "banana-forty-" appeared, find what followed it, and repeat that. Induction heads are this second kind of copying: not "what usually follows this" but "what followed this specific token last time, right here."

The two-head circuit

An induction head is not one head doing everything — it is a two-step circuit, usually split across two layers, found by opening up trained transformers and tracing which heads feed which.

Step 1 — the previous-token head (an earlier layer): at every position, attend to the token immediately before it, and copy that information forward.

Step 2 — the induction head (a later layer): at the current position (say, the second "Zorblax"), search back for earlier positions whose previous-token information matches the current token. Having found the earlier "Zorblax," attend to whatever came right after it, and copy that forward as the prediction.

$$\text{score}(i, j) = q_i^\top k_j, \qquad j^\star = \arg\max_j\ \text{score}(i,j)$$

In words: the induction head compares its query at position $i$ against keys $k_j$ built from "the token that followed an earlier occurrence of the current token," and picks the earlier position $j^\star$ scoring highest — the mechanism that finds "last time this token appeared, X came next."

Worked example 1: tracing the circuit on a toy sequence

Sequence: A B C A B C A ? — positions 1–7, predicting position 8. The pattern A B C repeats. At position 7 (token A), the earlier-layer head has already marked positions 1 and 4 as "preceded by A" (both are followed by B). The induction head at position 7 looks for earlier positions whose preceding token matches A — finds positions 2 and 5 — and copies what's there: B. Prediction: B, matching the true continuation. The circuit reconstructed the periodicity of a sequence it had never seen, using only attention lookups.

Worked example 2: measuring in-context learning with a loss curve

Researchers measure induction-head strength by plotting loss on repeated random token sequences as a function of position within the sequence, holding the model fixed. A model with strong induction heads shows a sharp drop in loss the moment a token repeats for the second time — because from then on, "look up what followed last time" solves the prediction outright.

Token position	Loss (nats)
1st occurrence of pattern	4.1
2nd occurrence (induction kicks in)	1.3
3rd occurrence	0.9

The steep drop between the 1st and 2nd occurrence — well before the model has "learned" anything about this specific random sequence in the weight-update sense — is the empirical signature of the circuit firing.

Drag the rate above — the sharp loss drop in the table behaves like a fast decay once the pattern has been seen once, similar in shape to this curve falling once its argument crosses a threshold.

What this means in practice

In-context learning driven by induction heads is why few-shot prompting works: showing a model two or three examples of a task format lets induction-like circuits copy "input → output" onto a new input, without fine-tuning. It also explains a known failure mode: models latch onto a superficial repeated pattern (a stray formatting quirk repeated twice) and copy it inappropriately, because the circuit doesn't know or care why the pattern held.

Practice

1. Sequence: X Y X Y X ?. Trace the two-step circuit by hand to predict the next token.
2. Why would an induction head fail on the sequence A B C D E A B C D ? if it can only look one step back for the "previous token" match, versus a version that matches on the last two tokens?