Choosing the Operating Point

A fraud model outputs a number between 0 and 1 for every transaction, but a bank doesn't block transactions with "a number" — it blocks transactions above some cutoff. That cutoff, called the operating point, is a decision with real consequences: set it too low and good customers get blocked constantly; set it too high and fraud sails through. Most people accept whatever cutoff their library defaults to, usually 0.5, without asking whether 0.5 relates to the actual costs of the two kinds of mistakes. Choosing the operating point turns a model's raw probability output into an actual policy.

Why 0.5 is usually the wrong answer

A classifier trained to predict, say, "will this trade be adversely selected" produces a probability $\hat{p}$ for each new trade. The 0.5 default implicitly assumes that a false positive (skipping a trade that would have been fine) and a false negative (executing a trade that gets adversely selected) cost exactly the same, and that positives and negatives are roughly balanced in the data. Neither is usually true. If adverse selection costs five times more than a missed opportunity, the threshold that minimizes expected cost is lower than 0.5 — you'd rather flag a trade too eagerly and occasionally skip a fine one than let a costly one through. The operating point should be derived from the actual cost ratio, not inherited from a library default:

$$\hat{p}^{*} \approx \frac{C_{FP}}{C_{FP} + C_{FN}}$$

In plain English: the optimal cutoff sits at the point where the cost of a false positive relative to the total cost of both error types balances out — if false negatives are far more expensive, the threshold drops, flagging more cases as positive even at the price of more false alarms.

Worked example: tuning a threshold from a cost table

A model flags trades for manual review. A false positive (flagging a fine trade) costs an analyst 10 minutes, valued at $8. A false negative (missing an adversely selected trade) costs an average of $400 in slippage. Plugging into the formula: $\hat{p}^{*} = 8 / (8 + 400) \approx 0.02$. With such lopsided costs, the model should flag almost anything with even a 2% predicted probability of adverse selection, since the cost of over-flagging is trivial next to the cost of missing one. Running the validation predictions at threshold 0.02 instead of 0.50 might raise the flagging rate from 3% of trades to 40% — a huge behavioral shift driven entirely by a cost ratio the default threshold ignored. A milder case — a marketing model where a false positive costs $5 and a false negative costs $50 — gives $\hat{p}^{*} = 5/55 \approx 0.09$, still below 0.5 but far less extreme, because the cost asymmetry is smaller.

Use the explorer above to see how a classifier's decision boundary — and therefore which points get called positive versus negative — shifts as you push the implied threshold around; it's the same idea as moving the cutoff along the probability axis rather than along a two-dimensional plane.

What this means in practice

Operating-point selection turns a model into a usable system: it's the last mile between "the model is 78% accurate" and "here is what the desk should actually do." The right threshold comes from a cost table (or a constraint, like "flag no more than 5% of trades for review") applied to the model's ROC or precision-recall curve, never from a default. Different deployments of the same model can legitimately use different thresholds if their cost structures differ.