ROC Curves and AUC

A model outputs a probability that a trade will be profitable, not a yes/no answer. To turn that probability into a decision, you pick a threshold — flag anything above 0.6 as "trade it," say — but the choice of threshold is arbitrary, and a single accuracy number computed at one threshold hides how the model behaves at every other cutoff you might have chosen instead. A model might look mediocre at threshold 0.5 but excellent at 0.7. The ROC curve solves this by showing performance across every threshold at once, and AUC condenses that whole curve into one comparable number.

An analogy: a smoke detector's sensitivity dial

Imagine a smoke detector with an adjustable sensitivity dial. Turn it up to maximum and it catches every real fire but also goes off from burnt toast constantly. Turn it down and false alarms vanish, but it might miss a real fire. There's no single "correct" setting — it depends on how costly a missed fire is versus an annoying false alarm. The ROC curve is what you'd get testing the detector at every possible dial setting, plotting the fraction of real fires caught against the fraction of false alarms triggered.

The two rates, and the curve they trace

At any given threshold, define the true positive rate, $TPR = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}}$ — the fraction of actual positives the model correctly flags — and the false positive rate, $FPR = \frac{\text{false positives}}{\text{false positives} + \text{true negatives}}$ — the fraction of actual negatives the model incorrectly flags. In plain English: TPR is "how many real fires did we catch," and FPR is "how many false alarms did we trigger, out of everything that wasn't actually a fire."

Sweep the threshold from 1 (flag nothing) down to 0 (flag everything) and plot $FPR$ against $TPR$ at every threshold: that trace is the ROC curve. A model with no predictive power produces a diagonal line from $(0,0)$ to $(1,1)$, catching positives and negatives in the same proportion — exactly what random guessing would produce. A useful model bows up and to the left, catching a high fraction of true positives while keeping false positives low. AUC, the area under that curve, summarizes the shape in one number between 0.5 (no better than random) and 1.0 (perfect ranking): it equals the probability the model scores a random true positive higher than a random true negative.

Worked example: ranking 5 trades

Suppose a model scores 5 candidate trades, 3 of which were actually profitable (P) and 2 were not (N), with scores $0.9(P), 0.8(N), 0.7(P), 0.4(N), 0.3(P)$. AUC by the "probability of correct ranking" definition counts, across all $3 \times 2 = 6$ possible (profitable, unprofitable) pairs, how many are ranked correctly (profitable scored higher): $(0.9,0.8)$ correct, $(0.9,0.4)$ correct, $(0.7,0.8)$ wrong (the unprofitable trade scored higher), $(0.7,0.4)$ correct, $(0.3,0.8)$ wrong, $(0.3,0.4)$ wrong. That's 3 correct out of 6, so:

$$AUC = \frac{3}{6} = 0.5 .$$

Despite the top score belonging to a genuinely profitable trade, the model's overall ranking is no better than a coin flip — the two low-scoring profitable trades drag it down, which a single "did it get the top one right" glance would have missed entirely.

What this means in practice

AUC is threshold-independent, which makes it good for comparing two models or two feature sets during research, before you've committed to a specific trading rule. But AUC treats every part of the ranking as equally important, which is often wrong in finance: you may only care about correctly ranking the top 5% of candidate trades, and a model can have high AUC while being mediocre exactly there. It's also insensitive to class imbalance in a way that can be misleading — a model with high AUC on a dataset where profitable trades are rare can still produce a poor precision at any threshold you'd actually deploy.