McNemar's Test for Comparing Classifiers

A new model gets 91% accuracy on the test set; the old model it's replacing gets 89%. Is that two-point gap a real improvement, or could it just as easily have come from noise in which particular test cases happened to land in the test set? Comparing two accuracy percentages directly, without a test, invites exactly this false confidence. When both models were scored on the same test set, McNemar's test gives a clean answer, because it uses the fact that the two models saw identical cases and can be compared case by case rather than as two independent summary statistics.

Why comparing the two disagreement cells is the right move

Build a 2x2 table not of predicted-versus-actual, but of model-A-correct versus model-B-correct across the same test cases:

	Model B correct	Model B wrong
Model A correct	$n_{11}$	$n_{10}$
Model A wrong	$n_{01}$	$n_{00}$

The cases where both models agree ($n_{11}$ and $n_{00}$) carry no information about which model is better — they'd cancel out of any comparison. All the evidence lives in the disagreement cells, $n_{10}$ (A right, B wrong) and $n_{01}$ (A wrong, B right). McNemar's test asks whether that split of disagreements is far from the 50/50 split expected if the two models were equally good:

$$\chi^2 = \frac{(|n_{10} - n_{01}| - 1)^2}{n_{10} + n_{01}}$$

In plain English: if the two models disagree about equally often in each direction, there's no evidence one is better; the test statistic grows only when the disagreements are lopsided toward one model consistently beating the other on the same cases.

Worked example: is the 2-point gap real?

On 1,000 shared test cases, both models agree 850 times (correct or wrong together, split however between $n_{11}$ and $n_{00}$, irrelevant to the test). Among the 150 disagreements, model A is right and B is wrong 95 times ($n_{10}=95$), and B is right and A is wrong 55 times ($n_{01}=55$). Applying the formula:

$$\chi^2 = \frac{(|95-55|-1)^2}{95+55} = \frac{39^2}{150} = \frac{1521}{150} \approx 10.1$$

A chi-squared statistic of 10.1 with 1 degree of freedom corresponds to a p-value well under 0.01 — strong evidence model A's edge isn't a coincidence. Contrast this with $n_{10}=78$, $n_{01}=72$ (the same 150 disagreements, nearly evenly split): $\chi^2 = (6-1)^2/150 \approx 0.17$, nowhere near significant — here the same two-point accuracy gap really could be noise.

What this means in practice

Use McNemar's test whenever two models are compared on one shared, fixed test set — champion-versus-challenger decisions, A/B comparisons against production, or a paper's claim of beating a baseline. It focuses statistical power on the cases that actually distinguish the two models, avoiding the common trap of eyeballing a small accuracy gap and calling it meaningful.