Instruction Tuning and Supervised Fine-Tuning

A model pretrained only to predict the next token on a huge scrape of internet text has learned an enormous amount about language and the world, but it has not learned to answer questions — asked "What is the capital of France?", a raw pretrained model is just as likely to continue with another question, or a Wikipedia-style tangent, because all of those are plausible continuations of that string somewhere in training data. It was trained to predict what comes next, not to be helpful. Instruction tuning closes that gap: take the pretrained model and continue training it — ordinary supervised learning, the same cross-entropy loss as pretraining — on a curated dataset of (instruction, ideal response) pairs, so "plausible continuation" and "helpful response" start to coincide.

The analogy: a widely read person learning to actually answer the question asked

Picture someone who has read an enormous, unstructured pile of books, articles, and transcripts — fluent on almost any topic, but never explicitly taught the convention "when asked a direct question, give a direct, on-topic answer, in that format." Instruction tuning is a focused course afterward, using worked examples that demonstrate: here is a request, here is what a good, direct response looks like — repeated across many instruction types until "request → helpful, on-format response" generalises.

The mechanics: same loss, different data

Pretraining and instruction tuning use the identical training objective — next-token cross-entropy loss — the entire difference is what data the loss is computed on.

$$\mathcal{L} = -\sum_{t} \log P(y_t \mid y_{<t}, \text{instruction})$$

In words: for each token $y_t$ in the response portion, the model is penalised by how much probability it assigned to the correct next token, given everything before it including the instruction — exactly the pretraining loss, computed only over response tokens (the instruction is typically masked out so the model isn't trained to "predict" a question it's given, only to predict good answers).

Supervised fine-tuning (SFT) data is far smaller than pretraining data — tens of thousands to low millions of examples versus trillions of tokens — but far more curated: written or reviewed by humans specifically to model the target behaviour, rather than being whatever happened to appear on the internet.

Worked example 1: before-and-after on one prompt

Prompt: "Summarize the plot of a mystery novel in two sentences." A pretrained-only model, continuing plausibly, might produce: "This is a common assignment in high school English classes..." — a plausible continuation as it might appear in a forum post, but not a summary. After instruction tuning on examples pairing instructions like this with actual two-sentence summaries, the same model given the identical prompt is far more likely to produce an actual summary, because the fine-tuning distribution consistently rewarded exactly that response shape for exactly that instruction.

Worked example 2: loss over an SFT training example

Instruction: "Translate 'good morning' to French." Target response: "Bonjour." Tokenised response: [Bon, jour, .]. During SFT, the loss is computed only over these 3 tokens, conditioned on the full instruction preceding them:

$$\mathcal{L} = -\log P(\texttt{Bon} \mid \text{instr}) - \log P(\texttt{jour} \mid \text{instr}, \texttt{Bon}) - \log P(\texttt{.} \mid \text{instr}, \texttt{Bon}, \texttt{jour})$$

If the pretrained model initially assigns only 5% probability to Bon as the first response token (uncertain this is a translation task), that token contributes a large loss term, $-\log(0.05) \approx 3.0$ nats, and gradient descent raises that probability specifically in instruction-following contexts like this one. Repeated across thousands of varied instruction types, this steadily reshapes the model's default behaviour toward direct, on-task responses.

next-token prediction huge, unlabelled, general Stage 2: instruction tuning small, curated (instruction, response) pairs

Think of the model's probability of "responding helpfully, on-format" as this S-curve as a function of how much instruction-tuning data it has seen — near zero for a purely pretrained model, rising sharply once enough instruction examples have reshaped the relevant behaviour.

What this means in practice

Instruction tuning is almost always the first of several post-training stages in a modern pipeline, typically followed by preference-based fine-tuning (RLHF or similar) that further shapes response style and safety beyond what supervised examples alone capture. For quant and finance applications, instruction-tuned models are the ones actually usable for tasks like summarising filings on command — a purely pretrained base model would need extensive prompt engineering to coax out similar behaviour, if achievable at all.

Practice

1. Why is the loss on an SFT example typically masked out for the instruction tokens and computed only over the response tokens?
2. A team fine-tunes a pretrained model on 10,000 instruction-response pairs about company earnings calls, hoping it will "learn" the specific historical earnings numbers used as examples. Explain in one or two sentences why this is likely to fail as a knowledge-injection strategy, even though the fine-tuning loss goes down.