Label Shuffling and Null Models

Any model, given enough flexibility, can find some pattern in a dataset even when the labels are pure noise with no genuine relationship to the features — this is one form of overfitting. A label-shuffling test (a type of permutation test) checks for this directly: shuffle the target labels randomly, so any true relationship between features and labels is destroyed, then retrain the model and measure its performance on this scrambled version many times over.

Repeating the shuffle-and-retrain process many times (say, 100 or 1,000 times) builds a distribution of scores achievable with no real signal present. Comparing the model's actual score on the true labels to that distribution answers the question a single accuracy number can't: is this performance meaningfully better than what an overfit model would achieve on pure noise, given this exact sample size and model complexity?

Worked example. A model trained to predict which stocks will beat the market next quarter achieves 58% accuracy on real labels. Shuffling the labels 200 times and retraining each time produces a distribution of "null" accuracies centered around 51%, with a 95th percentile of 56%. Because the real 58% score exceeds that 95th percentile, the result looks like genuine signal rather than an artifact of the model's flexibility.

This test is especially important with small samples or highly flexible models — both of which are common in finance — because that's exactly where a model is most able to overfit noise into a score that looks impressive but has no genuine predictive content.