Chinchilla Compute-Optimal Scaling

Before 2022, the dominant belief in large language model training was that bigger models were straightforwardly better, and the field kept scaling parameter count aggressively — GPT-3 at 175 billion parameters was trained on around 300 billion tokens, under 2 tokens per parameter. The Chinchilla paper trained over 400 models of varying sizes on varying data amounts, all under matched compute budgets, asking a sharper question than "does bigger help": for a fixed compute budget, what combination of model size and training data minimises loss? The answer overturned prevailing practice — the largest contemporary models were substantially undertrained, and a smaller model trained on far more data would have used the same compute more effectively.

The analogy: a fixed study budget split between textbooks and study hours

Imagine a student has a fixed number of hours to prepare for an exam, splittable between buying more textbooks (more parameters — more capacity) or spending more hours reading them (more training tokens — more exposure to data). A student who buys ten textbooks but only skims each will likely do worse than one who buys three and reads each thoroughly, given the same total hours. Chinchilla found the field had been buying too many textbooks and not reading them enough: model capacity was outpacing the data used to fill it.

The scaling relationship

Chinchilla modelled loss as a function of model size $N$ (parameters) and training tokens $D$, with a fixed compute budget $C \approx 6ND$ (a standard approximation for transformer training FLOPs):

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

In words: loss $L$ has an irreducible floor $E$, plus a term shrinking as model size $N$ grows (rate $\alpha$), plus a term shrinking as training tokens $D$ grows (rate $\beta$). Minimising this subject to a fixed budget $C=6ND$ gives an optimal split — and the fitted exponents were close enough that Chinchilla's headline rule of thumb is: scale $N$ and $D$ in roughly equal proportion — doubling compute should roughly double both, not just model size.

Worked example 1: comparing two allocations at equal compute

Fix compute budget $C = 6ND$ constant. GPT-3-style allocation: $N = 175\text{B}$, giving $D = C/(6N)$. Suppose $C \approx 3.1 \times 10^{23}$ FLOPs (roughly GPT-3's reported training compute). Solving for $D$: $D = 3.1\times10^{23}/(6\times175\times10^9) \approx 295$ billion tokens — close to what GPT-3 actually used.

Chinchilla's compute-optimal allocation for the same budget $C$ found roughly $N \approx 70\text{B}$, $D \approx 1.4$ trillion tokens — a smaller model, trained on nearly 5× more data, using the identical compute. Chinchilla (70B, compute-optimal) outperformed Gopher (280B, undertrained on the same-order compute) on downstream benchmarks despite having 4× fewer parameters — direct empirical confirmation that the reallocation, not raw size, was what mattered.

Worked example 2: applying the rule of thumb

The commonly cited simplified Chinchilla ratio is roughly 20 training tokens per parameter. For a target model of $N=13$ billion parameters, compute-optimal data would be approximately $D \approx 20 \times 13\text{B} = 260$ billion tokens. Training that same 13B model on only 50 billion tokens (under 4 per parameter, GPT-3-era practice) leaves it "big enough" but never seeing enough data to use its capacity fully — Chinchilla's fitted curves predict meaningfully higher final loss than a differently-sized model using the same compute more optimally.

Each loss term above falls as a power law in its respective variable ($N^{-\alpha}$, $D^{-\beta}$) — drag the exponent and watch how sharply diminishing returns set in; that shape is exactly what makes an equal split between more parameters and more data outperform pouring the whole compute budget into parameters alone.

What this means in practice

Chinchilla scaling reshaped how labs allocate training compute: rather than chasing ever-larger parameter counts, most modern frontier models are trained on token counts well beyond the naive "20× parameters" ratio, especially once inference cost is factored in — a smaller, more thoroughly trained model is also cheaper to run at scale, which matters enormously for production serving cost, a consideration Chinchilla's original loss-only framing did not directly optimise for.

Practice

1. If compute budget doubles and the Chinchilla ratio is held fixed, roughly how should both $N$ and $D$ change?
2. A team has a fixed compute budget and plans to serve their model billions of times after training. Explain in one sentence why they might rationally choose an allocation that is smaller-$N$ and larger-$D$ than the pure loss-minimising Chinchilla point.