Model fits for k-fold CV with a hyperparameter grid

Question

You use 5-fold cross-validation to choose among 8 candidate hyperparameter values. For each candidate you run the full 5-fold procedure, refitting the model on every fold. How many model fits happen in total during this cross-validation search (before any final refit)?

Accepted Answer

For a single hyperparameter value, 5-fold CV trains the model 5 times (once per fold). You repeat that for each of the 8 candidates: 5 	imes 8 = 40 	ext{ model fits}. After the search you would typically add one final refit on all the data using the winning hyperparameter, bringing the total to 41, but the CV search itself is 40 fits. Takeaway. With a grid of g settings and k folds, a CV search costs k 	imes g model fits; here 5 	imes 8 = 40.

Model fits for k-fold CV with a hyperparameter grid

Your answer

More Statistics questions