When disaggregating is the wrong move

Question

You compare a new smart order router against the old one on fill quality (fraction of "good" fills), splitting by how many child orders each parent was broken into:

	Few children	Many children	Overall
New router	780/1000 (78%)	18/100 (18%)	798/1100 (72.5%)
Old router	80/100 (80%)	200/1000 (20%)	280/1100 (25.5%)

The old router looks better in both subgroups (80% vs 78% with few children, 20% vs 18% with many), yet the new router wins overall (72.5% vs 25.5%). Your instinct from Simpson's paradox is to trust the subgroups and prefer the old router.

Is that right? Which router is actually better?

Accepted Answer

The twist: the splitting variable is caused by the treatment. In the usual Simpson's setup you condition on a confounder (something present before the choice, like market regime). Here, the number of child orders is decided by the router itself: the smart new router recognizes easy parents and executes them in few slices, while sending hard parents into many slices. Child-count is therefore a consequence (a mediator) of the router, not a pre-existing condition. Conditioning on a mediator is a mistake: it strips out precisely the smart routing decisions that make the new router good. By holdin…

When disaggregating is the wrong move

Your answer

More Statistics questions