Great in-sample, great in CV, dead live, what happened?

Question

A model scores well in-sample and holds up under cross-validation, so you ship it. Live, it earns nothing. What kind of error is this, and how is it different from ordinary overfitting?

Accepted Answer

This is data leakage (often look-ahead bias): information that would not have been available at prediction time leaked into training and into your validation, so both look great while the live system, which truly cannot see the future, fails. How it differs from overfitting. Plain overfitting leaves a tell: in-sample good, out-of-sample bad. Leakage is nastier because it inflates the out-of-sample and cross-validation numbers too, since the "future" contaminated the whole evaluation. The validation was never really honest. Common sources. - Scaling or feature selection using statistics comput…

Great in-sample, great in CV, dead live, what happened?

Your answer

More Statistics questions