Precision-Recall Curves

Suppose only 2% of days show a genuine breakout signal you want to catch, and 98% are ordinary noise. A model that flags nothing at all is already 98% accurate, and even a model with a respectable-looking ROC curve can still be nearly useless in practice, because ROC's false-positive rate is measured against a huge pool of negatives — flagging a small absolute number of them barely moves that rate, even if it means your "signal" list is mostly noise. When the thing you're hunting for is rare, precision and recall — and the curve traced by their trade-off — are the metrics that actually reflect what deploying the model would feel like.

An analogy: a metal detector on a beach

Picture searching a beach with a metal detector for buried coins, where coins are rare compared to bottle caps and other junk. Recall is the fraction of all the coins actually buried on the beach that you dug up — turn the detector's sensitivity way up and you'll find nearly every coin, at the cost of digging a hundred holes for bottle caps. Precision is the fraction of things you actually dug up that turned out to be real coins — turn sensitivity down and nearly everything you dig up is a genuine coin, but you'll walk past coins buried a little deeper. Neither number alone tells the whole story; the curve traces out every possible sensitivity setting and what precision and recall you'd get at each one.

The two rates, precisely

$$\text{Precision} = \frac{\text{true positives}}{\text{true positives} + \text{false positives}}, \qquad \text{Recall} = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}} .$$

In plain English: precision asks "of everything I flagged, how much was actually right?" while recall asks "of everything that was actually real, how much did I catch?" Sweeping the classifier's threshold from strict to lenient traces out the precision-recall curve — recall on the x-axis, precision on the y-axis. Unlike the ROC curve's diagonal baseline, the "no skill" baseline here is a flat horizontal line at the base rate of positives in the data (2% in the breakout example), which makes the curve visually honest about how hard the rare-event problem actually is.

Worked example: a breakout-detection model

Out of 1,000 days, 20 are genuine breakouts. At a strict threshold, the model flags 15 days, 12 of which are real breakouts:

$$\text{Precision} = \frac{12}{15} = 0.80, \qquad \text{Recall} = \frac{12}{20} = 0.60 .$$

Loosen the threshold and the model now flags 60 days, catching 18 of the 20 real breakouts but also flagging 42 non-breakouts as candidates:

$$\text{Precision} = \frac{18}{60} = 0.30, \qquad \text{Recall} = \frac{18}{20} = 0.90 .$$

Moving from the strict to the loose threshold raised recall from 60% to 90% — catching 3 more real breakouts — but crashed precision from 80% down to 30%, meaning 7 out of every 10 flagged days are now false alarms. Whether that trade is worth it depends entirely on the cost of missing a breakout versus the cost of investigating a false one; the curve makes that trade-off visible across every threshold rather than committing to one.

What this means in practice

Whenever the outcome you're modeling is rare — genuine alpha signals, fraud, margin calls, tail events — precision-recall curves give a truer picture than ROC/AUC, because they don't dilute performance against a huge pool of easy negatives. The right operating point on the curve is a business decision, not a statistical one: it depends on the relative cost of a false alarm (wasted capital, wasted attention) against a missed detection (an unflagged blowup). Report the whole curve, or the area under it, rather than a single precision or recall number computed at one arbitrarily chosen threshold.