The overall test failed, but it "works for left-handed users"

Question

An experiment shows no overall effect. Reluctant to give up, the team slices the data 15 ways, by country, device, tenure, age band, and finds the treatment is "significant for tablet users in Canada." They propose shipping to that segment.

Explain why post-hoc subgroup findings are usually spurious, and how to handle subgroup analysis responsibly.

Accepted Answer

It's multiple testing wearing a disguise. Testing 15 subgroups at 5% each, with no real overall effect, gives roughly a 1 - 0.95^{15} \approx 54\% chance that at least one lights up by pure luck. The "significant" segment is almost certainly the noise winner of an unadvertised 15-way search, and the reported p-value for it ignores the 14 other slices, so it's meaningless. Slicing finely also shrinks each subgroup's sample, making its estimate noisier and more prone to extreme, unrepresentative results (the winner's-curse inflation again). Two extra traps. 1. The overall null makes it worse. W…

The overall test failed, but it "works for left-handed users"

Your answer

More Statistics questions