The Corrected Paired T-Test for Cross-Validation
A modified version of the standard paired t-test that adjusts for the fact that cross-validation folds overlap and share training data, which makes a naive t-test overstate how significant a difference between two models really is.
Prerequisites: Hypothesis Testing
Comparing two models by running k-fold cross-validation and applying an ordinary paired t-test to the per-fold score differences seems reasonable, but it systematically understates the true variance of the estimate. The problem is that cross-validation folds are not independent: each fold's training set overlaps heavily with every other fold's training set, so the per-fold scores are correlated in a way a standard t-test assumes away — and that false assumption of independence makes the test too eager to declare a difference "significant."
A plain paired t-test on cross-validation fold scores understates the true variance because the folds share overlapping training data; the corrected version inflates the variance estimate to account for that overlap, giving a more honest p-value.
The fix
The Nadeau-Bengio correction adjusts the variance term in the t-statistic by a factor that accounts for the ratio of test-set size to training-set size across the folds, effectively treating the folds as more correlated than a naive calculation would. The corrected test statistic is otherwise structured just like a standard paired t-test, but with this inflated variance in the denominator, which produces a smaller t-statistic and a larger, more honest p-value than the naive version.
Model comparison papers and practitioner benchmarks that skip this correction risk reporting "statistically significant" model improvements that would not survive a properly adjusted test.
Related concepts
Practice in interviews
Further reading
- Nadeau, C. and Bengio, Y., 'Inference for the Generalization Error' (2003)