Pretraining Objectives: MLM, CLM and Span Corruption

Labeled data is scarce and expensive — nobody has hand-tagged a billion sentences with "this means X." Raw text, on the other hand, is everywhere: books, filings, forum posts. Pretraining is the trick that turns raw, unlabeled text into millions of free labeled examples, by hiding a piece of a sentence and asking the model to predict it. The model is graded against the very text it came from, so no human ever has to write an answer key. The only design choice is what gets hidden, and that choice is the difference between a model built to understand text (fill in a blank) and a model built to generate it (guess what comes next).

The analogy: three kinds of practice test

A student preparing for reading comprehension can practice three different ways. In the first, someone blacks out a random word in the middle of a paragraph and the student has to guess it using the sentence before and after — good practice for deep understanding, bad practice for writing, since the student never practices producing text with nothing after it to lean on. In the second, someone covers everything after a certain point in an essay and the student has to guess what comes next, using only what came before — poor for cross-referencing but exactly the skill needed to write an essay one word at a time. In the third, someone blanks out a whole missing sentence or two, not just single words, and the student has to produce the missing chunk in one go — practice for filling in a gap of any length, which is what translation and summarization actually require.

These three practice regimes are, respectively, masked language modeling, causal language modeling, and span corruption.

The mechanism: three ways to build a self-supervised label

All three objectives share the same scoring rule: cross-entropy loss between the model's predicted probability for the true hidden word and 1 (certainty). If $p$ is the probability the model assigned to the correct token, the loss contribution is

$$L = -\log p$$

In words: if the model was almost certain and right ($p$ close to 1), the loss is close to zero; if the model was confident and wrong, $p$ is close to zero and $-\log p$ blows up, punishing overconfident mistakes hard. Every pretraining objective below just differs in which tokens get a label attached and what context is available when predicting them.

Masked language modeling (MLM, encoder-only, e.g. BERT): pick roughly 15% of tokens at random, replace most of them with a special [MASK] marker, and ask the model to predict the original token using the entire surrounding sentence, both directions. This needs a full, unmasked attention pattern, so it only makes sense for encoder-only models.

Causal language modeling (CLM, decoder-only, e.g. GPT): for every position, predict the next token using only the tokens before it. There's no masking of input tokens at all — every single token in the training text becomes a prediction target, using whatever came before it. This is a causal-masked attention pattern by construction, so it fits decoder-only models exactly.

Span corruption (encoder-decoder, e.g. T5): replace contiguous spans of one or more tokens with a single sentinel marker, feed the corrupted text into the encoder, and train the decoder to generate the missing spans, one after another, in order. This exercises exactly the skill an encoder-decoder needs: read a full (corrupted) input, then generate a variable-length output.

Worked example 1: scoring one masked prediction

Sentence: "rates rose sharply today." Mask "rose." Suppose the model outputs a probability distribution over its vocabulary and assigns $p = 0.20$ to the true word "rose" (competing against "fell," "held," "jumped," and thousands of others). The loss for this one prediction is

$$L = -\log(0.20) = 1.609$$

Now suppose after more training the model assigns $p = 0.65$ to "rose": $L = -\log(0.65) = 0.431$. The loss dropped by more than a factor of three for going from a fairly uncertain guess to a fairly confident correct one — this single number, averaged over millions of masked tokens, is literally what gets minimized during pretraining.

Worked example 2: counting training signal per sentence

Take the ten-token sentence "the fund cut its exposure to energy names sharply this week" and compare how many prediction targets each objective extracts from it. MLM with a 15% masking rate masks about $0.15 \times 10 \approx 1$–$2$ tokens, generating 1–2 labeled predictions from this one sentence. CLM predicts every token from its left context, generating 9 labeled predictions (all but the first token, which has no left context). Span corruption, masking one contiguous span of length 3, generates 1 generation target (the 3-token span, predicted as a short sequence) but the encoder still processes all 10 input positions to build context. CLM's "predict everything" design is a large part of why decoder-only pretraining is so data-efficient per sentence — nothing is thrown away, every token becomes a lesson.

What this means in practice

The pretraining objective is not swappable after the fact — a model trained with MLM has never practiced generating open-ended text left to right, and one trained with pure CLM has never practiced using future context, so each is naturally suited to different downstream jobs (classification and embeddings versus open-ended generation) without extra fine-tuning. Span corruption sits deliberately in between, which is why encoder-decoder models remain the standard choice for tasks with a clearly defined input and a clearly defined, differently-shaped output, like translating a filing or summarizing a call transcript.

Practice

1. For the sentence "the desk closed the position early," write out the CLM training targets and their left-context inputs for every token.
2. Why can't causal language modeling be trained on a bidirectionally-masked encoder architecture — trace the argument back to the attention mask.
3. If a fine-tuning task requires reading a document and then writing a variable-length answer that isn't a direct extract from the document, which of the three pretraining objectives most closely rehearses that skill, and why?