Stratified and Grouped Cross-Validation

Cross-validation splits data into folds to test a model on data it wasn't trained on, but plain random splitting can go wrong in two common ways. If one class is rare, a random split might put almost none of it in a given fold, making that fold's evaluation noisy or meaningless. If observations come in correlated groups — many rows per company, or many trades per day — a random split can put some of a group's rows in the training fold and others in the test fold, letting the model implicitly "see" that group during training and inflating its apparent accuracy.

Stratified splitting preserves the overall proportion of each class in every fold, so a dataset that is 5% fraud and 95% non-fraud produces folds that are each close to 5%/95%, rather than some folds accidentally containing almost no fraud cases at all.

Grouped splitting instead respects clusters of related rows: every observation belonging to the same group — say, every daily return for one specific stock — is assigned to only one fold, never split across folds.

Worked example. A model predicts earnings surprises using ten years of quarterly data for 500 companies. If a random split puts three quarters of Company X's history in training and one quarter in the test fold, the model may have already learned Company X's idiosyncratic patterns before being "tested" on it, overstating accuracy. Grouping the split by company — training on some companies entirely and testing on others entirely — gives a much more honest read on how the model generalizes to a company it has never seen.

These two techniques address different failure modes and are often combined: stratify on the label while grouping on the entity, so folds are both balanced and leak-free.