Repeated K-Fold and the Variance of CV Estimates

Running 5-fold cross-validation once and reporting "82% accuracy" makes that number sound fixed, like a physical constant of the model. It isn't. The way the data happened to get split into 5 folds is itself a random choice, and a different random split — same data, same model, different shuffle — would have given a slightly different number. Repeated k-fold cross-validation runs the whole k-fold procedure several times with fresh random splits, precisely to measure how much that single number could have wobbled by chance alone.

The analogy: polling the same population twice

If a pollster surveys 1,000 random voters and gets 52% support for a candidate, a second poll of a different random 1,000 voters from the same population won't land on exactly 52% again — maybe 50%, maybe 54% — purely from which people happened to get sampled. No pollster reports one poll's number as if it were the exact truth; they report it with a margin of error built from imagining the poll repeated many times. Repeated k-fold does exactly that for a model's cross-validated accuracy: it repeats the "poll" (the fold split) many times so the spread of results is measured, not guessed at.

Worked example 1: one run vs. ten runs

A model scores 5-fold cross-validation once and gets accuracies of $[0.81, 0.84, 0.79, 0.83, 0.80]$ across the 5 folds, averaging to $0.814$. Reported alone, this looks precise. Now repeat the entire 5-fold procedure 10 times, each with a fresh random shuffle before splitting into folds. The 10 run-level averages come out as:

$$0.814,\ 0.798,\ 0.822,\ 0.805,\ 0.831,\ 0.797,\ 0.811,\ 0.819,\ 0.804,\ 0.826$$

The overall mean across all 50 fold-scores is about $0.813$ — close to the single run's $0.814$ — but the run-to-run spread ranges from $0.797$ to $0.831$, a swing of $3.4$ percentage points that a single run would never reveal.

Worked example 2: is Model A really better than Model B?

Model A averages $0.814$ across one 5-fold run; Model B averages $0.802$. A single run makes A look clearly better by $1.2$ points. Repeat both models across the same 10 random splits used above. Model A's 10 run-averages have a standard deviation of about $0.011$; Model B's have a standard deviation of about $0.013$. A rough standard error on the difference between the two run-averages is:

$$SE_{\text{diff}} = \sqrt{\frac{0.011^2}{10} + \frac{0.013^2}{10}} \approx 0.0054$$

The observed gap of $0.012$ is only about $2.2$ standard errors — suggestive, not the slam-dunk a single run's headline numbers implied. Without the repeats, this uncertainty would have been invisible, and a coin-flip difference between two models could easily have been reported as a real win.

Drag the parent distribution above and watch how the spread of sample means — analogous to the spread of run-level CV accuracies above — narrows or widens. The same logic applies here: each k-fold run's average accuracy is one draw from a distribution of possible run-averages, and repeating the run is how that distribution's spread gets estimated instead of assumed to be zero.

What this means in practice

Repeated k-fold (commonly 5 or 10 repeats of 5- or 10-fold CV) is standard whenever a model comparison or a headline accuracy number needs to be defensible, not just reported once and trusted. The extra compute cost is real — 10 repeats of 5-fold means training 50 models instead of 5 — but it is the only way to see the run-to-run variance that a single k-fold pass silently hides.