Learning Curves and Sample Complexity

You've built a model that predicts next-month stock returns and it isn't performing well enough. The obvious instinct is to go get more data — but is that actually the fix? If the model is already about as good as it will ever get given its structure, doubling your history will barely move the needle, wasting months of collection effort. A learning curve answers this directly: it shows what happens to error as you feed the model progressively more training examples, telling you whether you have a data problem or a model problem.

An analogy: studying for an exam with practice questions

Think of a student practicing for a standardized test. With only 5 practice questions, they can memorize the answers perfectly but do terribly on the real exam, because 5 questions don't cover the material. As they work through 50, then 500, then 5,000 questions, their practice-set performance gets slightly worse (harder to memorize everything) while their real-exam performance steadily improves, because they're learning patterns rather than memorizing answers. Eventually, more practice stops helping — the student has learned everything the material can teach them, and only better study material, not more of the same, would raise their score further.

The two curves, and what their gap means

Plot two lines against the training-set size $n$: the training error, measured on the data the model was fit to, and the validation error, measured on held-out data the model never saw. In plain English: training error typically starts near zero (a flexible model can fit a handful of points almost exactly) and rises as $n$ grows, since it's harder to fit every point with more of them. Validation error starts high and falls as $n$ grows, since the model is learning genuine patterns rather than the quirks of a tiny sample.

The gap between the two curves — and where they flatten — is the diagnosis:

• Both curves converge to a high error, close together: the model is too simple (high bias) for the pattern in the data. More rows of the same data will not help; you need a more flexible model or better features.
• A persistent gap between a low training error and a higher validation error: the model is overfitting (high variance). More training data typically does help here, because it narrows this gap by making it harder for the model to memorize noise.

Worked example: two models on the same signal

Suppose you're predicting whether a stock outperforms next week, and you track validation accuracy as training size grows from 500 to 8,000 labeled weeks.

Training rows	Model A validation accuracy	Model B validation accuracy
500	51%	53%
2,000	52%	58%
8,000	52%	63%, still rising

Model A plateaus almost immediately at 52%, and adding data from 2,000 to 8,000 rows changes nothing — it has hit its sample-complexity ceiling, and the fix is a richer feature set or a different model, not more rows. Model B keeps improving all the way to 8,000 rows and hasn't flattened yet, meaning it is data-starved: buying more historical observations for Model B likely keeps paying off, while doing the same for Model A would be wasted effort.

What this means in practice

Before spending a research budget on acquiring more history, alternative data, or a longer backtest window, plot a learning curve on what you already have. A curve that has already flattened tells you the honest next step is a better model, better features, or accepting the current ceiling — not more rows. A curve still falling at your current data size tells you that more data is probably the single highest-leverage thing you can do.