Cost-Sensitive Evaluation With Asymmetric Errors

Imagine two credit models with identical accuracy of 92%. Model A makes its mistakes mostly by rejecting good borrowers; model B makes its mistakes mostly by approving loans that later default. If a default costs the bank $20,000 and a rejected good customer costs $200 in lost fee income, these two "equally accurate" models are nowhere close to equally good — model B is a financial disaster and accuracy alone can't tell you that. Cost-sensitive evaluation replaces accuracy (which silently treats every mistake as equally bad) with a metric built directly from the actual dollar costs of each error type.

Building the metric

Start from the confusion matrix — true positives, false positives, true negatives, false negatives — and attach a cost to each cell instead of just counting it. The total expected cost of a classifier's predictions on a batch of $n$ cases is:

$$\text{Total Cost} = FP \times C_{FP} + FN \times C_{FN}$$

where $FP$ and $FN$ are the counts of false positives and false negatives, and $C_{FP}$, $C_{FN}$ are their respective dollar costs (true positives and true negatives are usually treated as costing nothing extra, though they can carry a benefit term too). In plain English: instead of asking "how many mistakes did the model make," cost-sensitive evaluation asks "how much did the mistakes it made actually cost," which is a completely different ranking whenever the two error types aren't equally expensive.

Worked example: re-ranking two models by cost, not accuracy

On a 1,000-loan validation set with a 5% default rate, Model A produces 10 false positives (rejects good borrowers) and 40 false negatives (approves defaulters); Model B produces 60 false positives and 15 false negatives. Both hit roughly 92-95% accuracy. Using C_{FP} = \200$and$C_{FN} = $20{,}000$:

Model A's cost: $10 \times 200 + 40 \times 20{,}000 = 2{,}000 + 800{,}000 = 802{,}000$, i.e. $802,000. Model B's cost: $60 \times 200 + 15 \times 20{,}000 = 12{,}000 + 300{,}000 = 312{,}000$, i.e. $312,000.

Despite similar or even better raw accuracy, Model A costs the bank two and a half times more than Model B, because it lets through more of the expensive error type. An accuracy leaderboard would never surface this; a cost-based leaderboard flags it immediately.

What this means in practice

Cost-sensitive evaluation is what actually determines which model gets deployed once real dollar amounts are at stake — accuracy, precision, and recall are useful diagnostics, but the deployment decision should run through an explicit cost table whenever the two error types differ in consequence, which in finance is almost always. It also feeds directly into threshold selection: once you have $C_{FP}$ and $C_{FN}$, the cost-minimizing operating point follows directly, and you can compare any two models — even ones built with entirely different architectures — on the single number that matters: expected dollars lost.