Train, Validation and Test Splits

Any model can be made to look brilliant on data it has already seen. Give a decision tree enough depth and it will reproduce every training row exactly, noise included, and report zero error. That number forecasts nothing. The only honest way to estimate how a model behaves on data it has never met is to hold some data back and genuinely refuse to look at it.

That much most people know. What gets missed is that you have to hold data back twice, for two different reasons, and conflating them is one of the most common mistakes in applied machine learning.

The analogy: past papers, the mock, and the real exam

A student preparing for an exam has three kinds of paper in front of her.

Past papers with the answers attached. She works through them, gets things wrong, and adjusts. This is the training set: the material the model is allowed to learn from directly.

A mock exam, sat under timed conditions. She does not memorise the mock's answers; she uses her score to decide how to prepare. This is the validation set. Notice what happens by her tenth mock: those scores stop being an honest prediction of the real exam, because she has been tuning herself against those particular papers.

The real exam. Sat once. Whatever comes out is the number that counts.

Three sets, three jobs, and they are not interchangeable:

Set	What it decides	How often you look
Train	The model's parameters — weights, tree splits, coefficients	Constantly, that is its purpose
Validation	Your choices — hyperparameters, features, which model family wins	Many times
Test	Nothing at all. It is only scored.	Once, at the very end

Cut by time, not at random

For data with no time axis, a random split is fine. For prices, order flow or anything else that arrives in sequence, a random split is a bug rather than a simplification. A shuffled split cheerfully trains on Wednesday and Friday and then tests on Thursday — and a market on Thursday looks an enormous amount like the two days bracketing it. The model needs no skill to score well, only the observation that neighbouring days resemble each other.

The gaps matter whenever a label looks forward. If you predict the return over the next five days, the last training row's label already covers the first five validation days; dropping a five-day buffer at each cut fixes it. Done systematically that is Purged & Embargoed Cross-Validation, part of the wider discipline in Cross-Validation for Time Series.

Worked example 1: why the validation score lies

Suppose you try 12 candidate models — three algorithms crossed with four hyperparameter settings. Suppose, uncomfortably, that all 12 are equally good, each with a true accuracy of exactly 60 percent, and that your validation set has 2,000 rows.

Each model's measured validation accuracy is a noisy version of 60 percent. The standard error of an accuracy estimate on $n$ rows is

$$\sqrt{\frac{p(1-p)}{n}} = \sqrt{\frac{0.60 \times 0.40}{2000}} = 0.0110$$

In words: each measurement wobbles about 1.1 percentage points around the truth, purely from which rows happened to land in the validation set.

Now pick the best of the 12. The expected maximum of 12 independent draws from a bell curve sits about 1.63 standard deviations above the centre, so the winner's measured score is roughly

$$0.600 + 1.63 \times 0.0110 = 0.618$$

You will write 61.8 percent in your report and the honest number is 60.0 percent. Every one of those 1.8 points is selection, not skill — you did not find a better model, you found a luckier validation set. Score the winner on the untouched test set and it comes back around 60 percent, which is why the test set exists at all. Try 100 candidates instead of 12 and the inflation grows to about 2.7 points.

Worked example 2: how big does the test set have to be?

You test a model on 500 rows and it scores 60 percent. How much does that pin down? Same formula:

$$\sqrt{\frac{0.60 \times 0.40}{500}} = 0.0219$$

A 95 percent interval is about $\pm 1.96 \times 0.0219 = \pm 0.043$, so the truth lies somewhere in 55.7 to 64.3 percent. A model testing at 60 percent and a rival testing at 63 are, on this evidence, indistinguishable.

Quadruple the test set to 2,000 rows and the standard error halves to 0.0110, giving $\pm 2.1$ points. Precision improves only with the square root of size, so halving the interval costs four times the data. That is the trade behind every split ratio, and it means 60/20/20 is a starting point rather than a law: what matters is the absolute count in each holdout, so 98/1/1 on five million rows still leaves 50,000 test rows and is plenty.