Decision Thresholds and Cost-Sensitive Learning

Your model looks at a transaction and returns 0.31. Now what? The number is not a decision. Somebody has to say whether 0.31 is high enough to block the card, and nothing inside the model answers that. In practice almost nobody decides — they call .predict(), the library silently compares against 0.5, and a business-critical choice gets made by a default argument.

That default is only correct in one situation: when being wrong in one direction costs exactly as much as being wrong in the other. Almost nothing in finance looks like that. Missing a fraudulent charge costs hundreds of dollars; wrongly flagging a good one costs a few minutes of an analyst's time. Any cutoff that treats those two as equal is throwing money away, and no amount of extra model training will fix it, because the problem is not in the model.

The analogy: the sensitivity dial on a smoke alarm

A smoke alarm has a detector and a dial. The detector measures particles in the air — that part is physics, and it is the same in every room. The dial sets how many particles trigger the siren.

In a kitchen you turn the dial down, because burnt toast waking the house is annoying but a missed fire is catastrophic. In a recording studio you might turn it up, because a false siren ruins a take. Same detector, same physics, different dial — because the consequences differ, not the smoke.

A classifier's probability output is the detector. The threshold is the dial. Retraining the model improves the detector. Choosing the threshold sets the dial. They are separate jobs and they are solved with separate information: the model is fitted from data, the threshold is chosen from costs.

The rule, in one line

Write $p$ for the model's estimated probability that the positive class is true — say, that this transaction is fraud. Write $C_{\text{FP}}$ for the cost of a false positive (you acted, and you should not have) and $C_{\text{FN}}$ for the cost of a false negative (you did nothing, and you should have acted).

If you act, you pay $C_{\text{FP}}$ whenever the case turns out negative, which happens with probability $1-p$. If you do not act, you pay $C_{\text{FN}}$ whenever it turns out positive, which happens with probability $p$. Acting is worth it exactly when its expected cost is smaller:

$$(1-p)\,C_{\text{FP}} \;<\; p\,C_{\text{FN}} \qquad \Longleftrightarrow \qquad p \;>\; t^{*} = \frac{C_{\text{FP}}}{C_{\text{FP}} + C_{\text{FN}}}$$

In words: the right cutoff is the false-alarm cost expressed as a share of the total cost of the two mistakes. If a false alarm is cheap relative to a miss, $t^{*}$ sits near zero and you act on flimsy evidence. If the two cost the same, $C_{\text{FP}} = C_{\text{FN}}$ and the fraction is exactly $1/2$ — which is where the library default comes from, and it is a claim about your costs, not a neutral position.

Worked example 1: card fraud, counted out

A bank scores 100,000 transactions. 300 of them are genuinely fraudulent — 0.3%. A missed fraud costs $400 in write-off and chargeback fees. A false flag costs $12 in analyst review and customer irritation.

Step 1 — the threshold. $t^{*} = 12 / (12 + 400) = 12/412 = 0.0291$. Act on anything above about 2.9%.

Step 2 — cost at the default cutoff of 0.5. At that height the model catches 120 frauds, misses 180, and raises 30 false flags.

$$\text{cost} = 180 \times 400 + 30 \times 12 = 72{,}000 + 360 = 72{,}360$$

Step 3 — cost at $t^{*} = 0.029$. Lowering the bar catches 255 frauds, misses 45, and raises 2,600 false flags.

$$\text{cost} = 45 \times 400 + 2{,}600 \times 12 = 18{,}000 + 31{,}200 = 49{,}200$$

That is $23,160 saved on 100,000 transactions, from changing one number. Now look at what happened to accuracy. At 0.5 the model got $100{,}000 - 180 - 30 = 99{,}790$ cases right, or 99.79%. At $t^{*}$ it got $100{,}000 - 45 - 2{,}600 = 97{,}355$ right, or 97.36%. Accuracy fell by 2.4 points while cost fell by a third. Accuracy is a scoreboard that prices every mistake at one dollar; your business does not.

Worked example 2: the same threshold logic on a trading signal

A model outputs $p$, the probability that a name rises over the next hour. The payoffs are not a cost matrix; they are a P&L. Enter the trade only if expected P&L clears the round-trip cost:

$$p\,g - (1-p)\,\ell - c > 0$$

where $g$ is the gain when you are right, $\ell$ the loss when you are wrong, and $c$ the round-trip transaction cost.

Case A — symmetric payoff. $g = 9$ bp, $\ell = 9$ bp, $c = 2$ bp. Then $9p - 9(1-p) - 2 > 0$, so $18p > 11$ and $p > 0.611$. Costs alone push the bar 11 points above the coin-flip line.

Case B — skewed payoff. Same model, but the trade is structured so a win pays $g = 14$ bp and a loss caps at $\ell = 6$ bp, with the same $c = 2$ bp. Then $14p - 6(1-p) - 2 > 0$, so $20p > 8$ and $p > 0.40$.

Now take a signal that fires at $p = 0.55$ and check both by hand. Under Case A the expected P&L is $0.55(9) - 0.45(9) - 2 = 4.95 - 4.05 - 2 = -1.10$ bp — a losing trade. Under Case B it is $0.55(14) - 0.45(6) - 2 = 7.70 - 2.70 - 2 = +3.00$ bp — a good one. Identical model, identical probability, opposite decisions, because the payoff geometry moved the threshold from 0.611 to 0.40.

Cost-sensitive learning: the other way to the same place

There are two ways to make a model care about asymmetric costs.

Move the threshold after training. Fit normally, then set $t^{*}$ from the cost ratio. Cheap, transparent, reversible.

Weight the training loss. Multiply the loss on positive examples by a factor $w$ so the optimiser tries harder not to miss them. For a well-calibrated model the two are the same thing. If you upweight positives by $w$, the model's reported probability $p_w$ relates to the honest probability $p$ by $p_w = wp / (wp + 1 - p)$, and thresholding $p_w$ at 0.5 means

$$wp > 1-p \qquad \Longleftrightarrow \qquad p > \frac{1}{1+w}$$

Check it against example 1. The cost ratio there was $400/12 = 33.3$, so $w = 33.3$ gives a threshold of $1/34.3 = 0.0292$ — the same 2.9% cutoff you got from the cost formula directly. The two roads meet.

The explorer below draws a logistic curve; drag its steepness and watch how many points sit close to any horizontal cutoff line. A flat curve means the threshold choice moves a huge number of cases, so it matters enormously. A steep one means most cases are far from any plausible cutoff and the threshold barely matters.

What this means in practice

The threshold is free alpha, and it is usually left on the table. Retraining to squeeze out another point of AUC is expensive. Recomputing a cutoff from a cost ratio takes an afternoon and often moves more money.

Calibration is a prerequisite, not a nicety. The formula reads the model's output as a real probability. If a gradient-boosted model's "0.03" actually means 8% in the wild, $t^{*}$ is being applied to a number that does not mean what the derivation assumed. Calibrate first, threshold second.

Costs drift. Analyst headcount changes, chargeback fees get renegotiated, execution costs fall. The threshold should be a configuration value that someone owns and revisits, not a constant compiled into a model artefact.

Tune it on validation data, report it on test data. Sweeping thresholds on the same set you quote results from is just Data-Snooping Bias wearing a different hat.

Practise it

1. A model flags loan defaults. A missed default costs $9,000; a wrongly declined good applicant costs $150 in lost margin. What is $t^{*}$? (Answer: $150/9150 = 0.0164$.)
2. Your risk team says false positives now cost $300 instead of $12 in example 1, because each flag freezes a customer account. Recompute $t^{*}$ and decide whether the default 0.5 is now closer to right. (Answer: $300/700 = 0.429$ — much closer, but still below 0.5.)
3. In example 2 Case A, what does the round-trip cost have to fall to before a $p = 0.55$ signal becomes profitable? (Answer: $c < 0.90$ bp.)
4. Take any model you have trained. Plot total cost against threshold using your real cost numbers and mark where 0.5 sits. If the curve is flat there, the default was harmless; if it is on a steep flank, you have been leaving money behind.