Knowledge Distillation

A senior trader can explain not just their trade, but how close a call it was — "I bought, but it was maybe 55/45 versus selling." A junior who only sees "buy" or "sell" written on a ticket learns far more slowly than one who hears the hesitation, the near-miss, the "I almost did the opposite." Knowledge distillation gives a small model that same richer signal: instead of learning only the correct label for each example, it learns the large model's full confidence distribution across every possible label, including the wrong ones.

A large, accurate model — the teacher — is expensive to run: too slow or too memory-hungry for production, live trading, or a mobile device. A student, a much smaller model, needs to approximate the teacher's behaviour cheaply. Training the student directly on the same labelled data the teacher used often falls well short of the teacher's accuracy, because the student has far less capacity to independently rediscover every pattern from raw labels alone. Distillation instead trains the student to match the teacher's output distribution.

Soft targets: the information hiding in "almost"

A classifier's raw output layer typically ends in a softmax, producing a probability for every class. A hard target — the label actually used for training with ordinary cross-entropy — collapses that to one number: $1$ for the true class, $0$ for everything else. A soft target keeps the teacher's full distribution: for an image that is mostly a cat, the teacher might output cat $0.7$, dog $0.25$, fox $0.04$, car $0.01$. That dog probability of $0.25$ is information: it says "this cat looks somewhat dog-like," a relationship the hard label $1/0/0/0$ throws away entirely.

Distillation controls how soft these targets are with a temperature $T$ applied inside the softmax:

$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

Here $z_i$ is the teacher's raw pre-softmax score ("logit") for class $i$, and $p_i$ is the resulting probability. In words: dividing every logit by $T$ before exponentiating shrinks the differences between them when $T > 1$, spreading probability mass more evenly across all classes — turning a confident $0.98/0.01/0.01$ output into something like $0.6/0.25/0.15$, which reveals the relative similarity between the wrong classes that a near-certain prediction hides. At $T=1$ you get the ordinary softmax; as $T \to \infty$ the distribution flattens toward uniform.

Drag the curve's steepness parameter and watch a sharp S-curve flatten into a gentle slope — that's the same effect temperature has on a softmax: a high, confident peak (steep curve) versus a spread-out, informative distribution (gentle curve) over the same set of classes.

The student is trained on a weighted combination of two losses: the usual cross-entropy against the true hard label, and a second cross-entropy against the teacher's softened ($T>1$) probabilities. The second term is what transmits the teacher's "hesitation."

Worked example 1: soft vs hard target for one example

An image is truly a "call option payoff" chart. The teacher's raw logits (before softmax) for three classes {call payoff, straddle payoff, put payoff} are $z = [4.0, 3.5, 0.2]$. At $T=1$: $\exp(4.0)=54.6$, $\exp(3.5)=33.1$, $\exp(0.2)=1.22$, sum $=88.9$, giving probabilities $\approx [0.614, 0.372, 0.014]$ — already fairly informative, since call and straddle are close. At $T=4$: divide logits by 4 first, $z/T = [1.0, 0.875, 0.05]$, then $\exp(1.0)=2.72$, $\exp(0.875)=2.40$, $\exp(0.05)=1.05$, sum $=6.17$, giving $\approx [0.441, 0.389, 0.170]$. The gap between "call" and "straddle" shrank from 24 points to 5 points of probability, and "put" — genuinely a much less similar shape — still trails clearly. The student now sees, in one training example, "this looks like a call, quite a bit like a straddle, and not much like a put," instead of just "this is a call."

Worked example 2: why the student can beat training on hard labels alone

Suppose a small student trained directly on 1,000 hard-labelled examples reaches 82% accuracy — it lacks capacity to fully separate every class from raw labels alone. The same architecture trained via distillation on the identical 1,000 images, using a large teacher's soft outputs, reaches 89%. The extra 7 points didn't come from new data; they came from the teacher compressing what it learned from millions of examples into the shape of its outputs on those same 1,000 images, and the student absorbing that shape instead of rediscovering class relationships from scratch.

What this means in practice

Distillation is how a quant firm gets a large, slow model — an ensemble, a big gradient-boosted forest — into something that can score thousands of instruments in real time. It's standard for compressing large language models into deployable assistants, and equally applicable to compressing a slow research ensemble into a fast, small-network approximation usable intraday.