Nested cross-validation, tuning without cheating

Question

You use k-fold cross-validation to pick your hyperparameters and then report that cross-validation score as your estimate of out-of-sample performance. Why is that number optimistic, and how does nested cross-validation fix it?

Accepted Answer

The problem. When you tune hyperparameters by trying many settings and keeping the one with the best cross-validation score, you have used the cross-validation to select. The winning score is the maximum over many noisy estimates, so it is biased upward, the same selection effect as tuning on a validation set. Reporting that best score as your generalization estimate double-counts the data: the folds that chose the model are also being asked to judge it. The model has, in effect, seen those folds during selection. The fix, nested cross-validation. Use two loops. - Inner loop: within each trai…

Nested cross-validation, tuning without cheating

Your answer

More Statistics questions