Subsample Stability Across Universe Slices

A full-sample IC of 0.02 across a universe of 3,000 names is one number that could arise from a hundred different underlying stories. It could be a genuinely broad effect present everywhere at a similar strength. It could also be a strong effect confined to tech stocks and roughly zero everywhere else, or an effect that only exists in the Russell 2000 and not the S&P 500. Subsample stability testing splits the universe along natural lines — sector, size, region, exchange — and re-measures the signal within each slice, to find out which story is true.

What you're looking for

The goal isn't a perfectly flat IC across every slice — that's an unrealistic bar and real effects are allowed to vary in strength. The goal is to rule out the case where the full-sample number is being carried entirely by one or two slices while the rest of the universe contributes nothing or works against it. A signal present, with the same sign, at varying strength across most slices is a broad effect. A signal that's strongly positive in one slice and near zero or negative elsewhere is a narrow effect wearing a broad-sample average as a disguise.

A worked example

A signal's full-sample IC across all GICS sectors is 0.018. Splitting by sector: ten of the eleven sectors show IC between 0.005 and 0.015, roughly consistent with sampling noise around a small positive effect. Technology alone shows an IC of 0.09. Recomputing the full-sample IC with technology excluded drops it to 0.008 — most of the headline number was one sector. That doesn't kill the idea, but it changes what gets reported and how it gets sized: this is a technology-sector effect, plausibly tradeable on a smaller book concentrated there, not the broad cross-sectional signal the original number implied.