K-Fold Cross-Validation

You have 200 days of labeled data — say, whether a stock beat its sector the next week — and you want an honest estimate of how a model will do on data it hasn't seen. Split the 200 days once into 160 training and 40 testing, and you get a single number that depends heavily on which 40 days happened to land in the test set: a calm stretch versus a volatile one can swing your accuracy estimate by several points, and you've thrown away information, since those 40 days never helped train the model. K-fold cross-validation fixes both problems by rotating which slice gets held out, so every observation eventually plays both roles.

An analogy: rotating who grades the essays

Imagine five study partners preparing for the same exam. Instead of one partner grading everyone's practice essays, the group splits into five smaller groups. Each round, four groups study together and write practice essays, and the fifth group — who studied nothing that round — grades them. Over five rounds, every group gets a turn grading and being graded, and you average the five scores into one estimate. No one's essays are graded by someone who helped write them.

The mechanics, one symbol at a time

Split the $n$ observations into $k$ roughly equal, non-overlapping groups called folds. For each fold $i = 1, \dots, k$: train the model on all data except fold $i$, then measure its error on fold $i$ alone — data the model never touched during that round. The cross-validated error is the average across all $k$ rounds:

$$CV_{(k)} = \frac{1}{k} \sum_{i=1}^{k} \text{Error}_i .$$

In plain English: instead of one train/test split producing one possibly-lucky or possibly-unlucky number, you get $k$ independent-ish estimates and average them, which is both more stable and uses every observation for testing exactly once.

Worked example: 10 days, 5 folds

Suppose you have 10 days of return-prediction errors and use $k=5$, so each fold holds out 2 days. Round 1 trains on days 3–10 and tests on days 1–2, getting a mean-squared error of $0.40$. Rounds 2 through 5 rotate the held-out pair and give errors of $0.55$, $0.35$, $0.60$, and $0.50$:

$$CV_{(5)} = \frac{0.40+0.55+0.35+0.60+0.50}{5} = 0.48 .$$

If by chance a single 80/20 split had put the two easiest days (behind the 0.35 result) in the test set, you'd have reported 0.35 and believed the model noticeably better than it is. Averaging across rotations smooths out that luck.

Choosing k

Larger $k$ (up to $k=n$, leave-one-out) trains on more data per round, giving a nearly unbiased estimate, but the resulting error estimates are highly correlated and expensive to compute. Smaller $k$ trains on less data each round, biasing the estimate upward, but runs fast. In practice $k=5$ or $k=10$ is the standard compromise.

What this means in practice

K-fold cross-validation is the default tool for choosing hyperparameters and getting an honest read on a model's generalization error before it ever sees live data. It's also the workhorse behind grid search: you run cross-validation once per candidate hyperparameter setting and pick the one with the lowest average error. But standard k-fold assumes each observation is roughly independent of the others — a plain random shuffle of time-ordered financial data lets a model trained on data from next week "peek" at patterns and test on last week, silently leaking information across the split.