Mechanistic Interpretability and Circuits

Most interpretability methods treat a network as a black box and only probe its inputs and outputs. Mechanistic interpretability takes the opposite approach: open the network up and reverse-engineer the actual algorithm running inside it, the way you'd reverse-engineer compiled code back into readable source. The unit of analysis is a circuit — a specific, small subset of neurons and connections (or, in transformers, attention heads) that together implement one identifiable piece of computation, like detecting a specific pattern or copying a value from one position to another.

Why bother

A surrogate model or a feature-attribution method gives you a summary of what a model does, averaged or approximated. Mechanistic interpretability aims for something stronger: a mechanistic, causal account of how — this specific neuron detects rising volatility because it fires on the difference between two upstream neurons that each track price range, and that neuron's output is read by a later layer that down-weights position size. If true, that account should let you predict the network's behavior on inputs you haven't tried yet, and edit the network's behavior surgically by changing just that circuit, rather than retraining the whole thing.

The circuits workflow

Researchers typically: (1) find a behavior of interest, such as the model reliably reducing exposure after three consecutive down days; (2) use activation patching — running the model on a "clean" input, saving a specific internal activation, then splicing it into a run on a "corrupted" input to see whether the behavior transfers — to localize which components causally matter; (3) narrow down to the smallest set of neurons or attention heads whose removal (ablation) kills the behavior; (4) write down, in plain language and ideally in code, exactly what that circuit computes, then verify the hand-written account reproduces its behavior on new inputs.

Worked example

A research team suspects a small transformer used for order-flow classification has learned a "reversal detector." Using activation patching, they find that zeroing out just two attention heads in layer 3 collapses reversal-flagging accuracy from 89% to 52% (near chance) while every other prediction task stays untouched, localizing the behavior to those two heads. Inspecting what the heads attend to, they find head 3-7 attends from the current bar back to the bar three steps prior whenever the sign of returns flips, and head 3-2 reads that pattern and writes a "reversal" signal downstream. They write this up as an explicit rule, test it against 500 held-out sequences, and find their hand-coded version agrees with the two heads' combined output 94% of the time — reasonably strong evidence the circuit implements a three-bar reversal check.

What this means in practice

For trading models, mechanistic interpretability is expensive and mostly reserved for high-stakes, hard-to-trust behaviors, since it requires deep access to internals and painstaking manual work per behavior rather than a one-line library call. When it succeeds, though, it gives something no attribution method does: a testable, falsifiable, human-written hypothesis for a mechanism, which can then be checked, monitored, or deliberately edited.