Why you need three data splits, not two

Question

Explain the roles of training, validation, and test sets. Why is two splits not enough once you start tuning a model, and what goes wrong if you make decisions on the test set?

Accepted Answer

Think of three jobs, and give each its own slice of data so no slice does two jobs at once. Training set. Used to fit the model: the parameters (weights, coefficients, tree splits) are learned here. A model always looks good on the exact data it was fit to, so training error alone tells you almost nothing about the future. Validation set. Used to choose: which features, which hyperparameters (regularization strength, tree depth, learning rate), which model family. You try many options, score each on the validation set, and keep the best. This is model selection, and it burns information: afte…

Why you need three data splits, not two

Your answer

More Statistics questions