McNemar's Test for Paired Proportions

Suppose you have two classifiers — say, two versions of a model predicting whether a stock beats earnings — and you run both on the same set of names. You want to know if one model is genuinely better, not just different on the overall hit rate, which can hide a lot. McNemar's test does this cleanly by looking only at the cases where the two models disagreed: cases where both were right or both were wrong tell you nothing about which is better, so they're discarded entirely.

Build a 2×2 table of disagreements: $n_{01}$ counts cases where model 1 was wrong and model 2 was right, and $n_{10}$ counts the reverse. Under the null hypothesis that neither model is systematically better, these two counts should be roughly equal. The test statistic is

$$\chi^2 = \frac{(n_{01} - n_{10})^2}{n_{01} + n_{10}} ,$$

compared against a chi-squared distribution with one degree of freedom. In plain English: it asks whether the disagreements lean lopsidedly toward one model, using only the disagreement counts and ignoring every case both models agreed on.

Worked example

Testing two earnings-surprise classifiers on 200 stocks: both right on 120, both wrong on 50, model 1 right where model 2 was wrong on 10 stocks ($n_{10}=10$), and model 2 right where model 1 was wrong on 20 stocks ($n_{01}=20$). Then

$$\chi^2 = \frac{(20-10)^2}{20+10} = \frac{100}{30} \approx 3.33 ,$$

which falls short of the standard 3.84 cutoff for significance at the 5% level with 1 degree of freedom — so despite model 2 winning twice as often on the disagreement cases, the sample of 30 disagreements isn't yet large enough to rule out that this lean is just noise.