Scaling Laws for Language Models
As you make a language model bigger, train it on more data, and give it more compute, its performance improves in a strikingly predictable way, and these scaling laws are what let researchers plan enormous training runs in advance.
A scaling law is an empirical rule showing that a language model's loss — how badly it predicts the next word — falls off as a smooth, predictable power-law curve as you increase model size, training data, or compute, each measured in orders of magnitude. Instead of guessing whether a bigger model will help, researchers can extrapolate along the curve and estimate performance before spending the money to train it.
Language model performance improves as a predictable power-law function of model size, data, and compute — which is why labs can forecast a giant training run's outcome in advance, and why the real design question became how to split a fixed compute budget between a bigger model and more data, not whether bigger is better.
The practical finding that reshaped the field is that model size and training data need to scale together. Early scaling work suggested pouring most of a compute budget into parameters; later work, the "Chinchilla" scaling laws, found that many large models had actually been undertrained relative to their size, and that a smaller model trained on proportionally more data reached better performance for the same compute cost.
Worked example. Two labs each have the compute budget to run 10^23 floating-point operations. Lab A spends it on a 175-billion-parameter model trained on a small dataset relative to its size. Lab B spends the same budget on a 70-billion-parameter model trained on several times more data. Chinchilla-style scaling laws predict Lab B's smaller, more data-fed model will actually outperform Lab A's undertrained giant, for identical training cost.
For quant and financial applications, the practical takeaway is less about training models from scratch and more about interpreting vendor claims: a model advertised mainly by parameter count, without disclosing its training data volume, is not obviously the stronger choice.
Related concepts
Practice in interviews
Further reading
- Kaplan et al., 'Scaling Laws for Neural Language Models'
- Hoffmann et al., 'Training Compute-Optimal Large Language Models' (Chinchilla)