The Confusion Matrix and Classification Metrics

A classifier hands you a verdict — fraud or clean, up or down, default or performing — and you want one number that says how good it is. Almost everybody reaches first for accuracy: what fraction of the calls were right. On a card book where one transaction in a hundred is fraudulent, a model that flags absolutely nothing scores 99 percent accuracy and is worth precisely nothing. Accuracy is not wrong. It is answering a question nobody asked.

The fix is to stop compressing to one number too early. Write down all four things that can happen, then build whichever ratio matches the decision you actually face.

The analogy: the bag scanner at the airport

A scanner looks at bags and either pulls one aside or waves it through. Exactly four things can happen and there is no fifth:

• it pulls a bag that really did have a knife in it — a catch
• it pulls a bag that was perfectly innocent — a false alarm
• it waves through a bag with a knife inside — a miss
• it waves through a harmless bag — a correct pass

Every argument about airport security is an argument about the trade between false alarms and misses. Turn the sensitivity up and you catch more knives and also search more grandmothers. Turn it down and the queue moves fast and something eventually gets through. No setting removes both. That trade is the entire subject; the metrics are just different ways of pricing it.

The four boxes

Machine learning gives the four outcomes uglier names. Read them in two halves: positive always means the model said yes, and true / false means the model was right / wrong.

• $TP$, true positives — flagged, and it really was the thing. The catch.
• $FP$, false positives — flagged, but it wasn't. The false alarm.
• $FN$, false negatives — not flagged, but it really was. The miss.
• $TN$, true negatives — not flagged, and it wasn't. The correct pass.

Arranged as a two-by-two table these are the confusion matrix, and every metric below is one ratio drawn from it. What separates the metrics is entirely which denominator they choose.

$$\text{Precision} = \frac{TP}{TP + FP}$$

In words: of everything the model flagged, what share was real? This is the question asked by whoever has to work the alert queue.

$$\text{Recall} = \frac{TP}{TP + FN}$$

In words: of everything that was really there, what share did the model find? This is the question asked by whoever is exposed to the misses.

$$F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

In words: a single score that is high only when both are high. It is the harmonic mean, which unlike a plain average gets dragged down hard by whichever of the two is smaller — 0.9 and 0.1 average to 0.5 but give an $F_1$ of 0.18.

Worked example 1: a fraud screen, every number computed

Ten thousand transactions. One hundred of them are genuinely fraudulent, so the base rate is 1 percent. The model flags 180 transactions, and 60 of those flags are real fraud.

From those three sentences the whole table follows. $TP = 60$. Flags that were wrong: $FP = 180 - 60 = 120$. Fraud that was missed: $FN = 100 - 60 = 40$. Everything else: $TN = 10{,}000 - 180 - 40 = 9{,}780$.

	model flags	model passes	row total
really fraud	60	40	100
really clean	120	9,780	9,900
column total	180	9,820	10,000

Now every metric, one line each:

• Accuracy $= (60 + 9780)/10000 = 0.984$ — right 98.4 percent of the time.
• Precision $= 60/180 = 0.333$ — one alert in three is real; the analyst wastes two reviews out of every three.
• Recall $= 60/100 = 0.600$ — three fifths of the fraud is caught, two fifths walks.
• F1 $= 2(0.333)(0.600)/(0.333 + 0.600) = 0.400/0.933 = 0.429$.
• Specificity $= TN/(TN+FP) = 9780/9900 = 0.988$ — of the clean transactions, 98.8 percent were correctly left alone.
• False positive rate $= 120/9900 = 0.012$ — 1.2 percent of clean traffic gets bothered.

Hold that 98.4 percent accuracy against the do-nothing baseline: flag no transaction at all and you are right on all 9,900 clean ones, an accuracy of 99.0 percent. The useless model beats the working one on the headline metric. Precision and recall, which never look at $TN$ at all, both correctly report zero for the do-nothing model.

Worked example 2: what moving the threshold does

Underneath, the model outputs a score and you flag anything above a cut-off. Lower that cut-off and you catch more of everything — real fraud and innocent traffic alike.

Drop the cut-off until the model flags 500 transactions instead of 180, and suppose that catches 85 of the 100 frauds. Then $TP = 85$, $FP = 415$, $FN = 15$, $TN = 9900 - 415 = 9485$.

• Accuracy falls: $(85 + 9485)/10000 = 0.957$.
• Recall jumps: $85/100 = 0.850$.
• Precision collapses: $85/500 = 0.170$.
• F1 falls: $2(0.170)(0.850)/(1.020) = 0.283$.

Three of the four metrics say the new threshold is worse. It is exactly the same model — not one weight changed — and you have moved a single number.

What this means in practice

The metrics disagree because none of them knows what an error costs you. Put money on it and the ambiguity vanishes. Say a missed fraud costs $500 in write-offs and a false alarm costs $8 of analyst time:

• Do nothing: $100 \times 500 = 50{,}000$, so $50,000.
• Threshold A (flag 180): $40 \times 500 + 120 \times 8 = 20{,}000 + 960$, so $20,960.
• Threshold B (flag 500): $15 \times 500 + 415 \times 8 = 7{,}500 + 3{,}320$, so $10,820.

Threshold B is worth roughly $10,000 a batch more than A, despite scoring lower on accuracy, precision and F1. Those metrics were silently assuming a false alarm and a miss hurt equally. Here a miss costs 62.5 times more, so the right operating point buys recall with precision until the marginal alarm costs as much as the fraud it prevents — see Cost-Sensitive Evaluation With Asymmetric Errors and Choosing the Operating Point.

Three habits follow. Report the raw four counts, not only ratios, so a reader can rebuild any metric you didn't think of. Separate the model from the threshold — ROC Curves and AUC and Precision-Recall Curves score the ranking across all thresholds, which is what you want when comparing models rather than deployments. And quote the base rate every time, because a precision of 0.33 is dismal at a 30 percent base rate and remarkable at 1 percent.