The Cochran-Mantel-Haenszel Test

A single 2x2 contingency table can badly mislead you if a third variable is lurking underneath it. Imagine testing whether a trading signal "works" (flagged trade profitable or not) pooling data from bull and bear markets together: if the signal fires more often in bull markets, and bull markets are simply more profitable regardless of any signal, a plain chi-squared test on the pooled table can show a fake association driven entirely by market regime, not the signal itself — this is Simpson's paradox in table form. The Cochran-Mantel-Haenszel (CMH) test fixes this by testing for association within each stratum (each regime, here) and combining the evidence across strata, rather than pooling the raw counts first.

Concretely, for each stratum $i$ with a 2x2 table of counts, CMH compares the observed count in one cell, $a_i$, against the count it would expect under independence within that stratum, then sums both the observed-minus-expected differences and their variances across all strata before forming a single chi-squared statistic. A stratum where the signal fires and profits move together in one direction, and another where they move together in the opposite direction, don't cancel each other out in a way that pooling would — each stratum's own within-group evidence is preserved and added.

The result is one overall test statistic, asymptotically chi-squared with 1 degree of freedom, for "is there an association between the two variables, common across strata, once you've controlled for the stratifying variable" — the categorical counterpart to adding a control variable in a regression.