The Base Rate Fallacy

A trading signal fires "regime shift alert" and back-testing shows it's correct 95% of the time it fires when a real regime shift is happening. The signal just fired. How confident should you be that a real regime shift is actually happening right now? Most people's gut answer is "about 95%." That answer is almost always wrong, and the reason has nothing to do with the signal's accuracy — it has to do with a number nobody mentioned yet: how often do real regime shifts even happen? If genuine regime shifts are rare — say, they occur on only 1% of days — then even a very accurate signal will, most of the time it fires, be firing on one of the vastly more numerous ordinary days that merely happened to look similar, not on an actual regime shift. Ignoring how rare the target event is in the first place is the base rate fallacy, and it's one of the most consistently underestimated traps in signal evaluation.

An analogy: a disease test in a healthy population

A medical test correctly flags a rare disease 99% of the time when someone actually has it, and correctly clears healthy people 99% of the time. The disease affects 1 in 10,000 people. You test positive. How worried should you be? Out of 10,000 random people, about 1 actually has the disease, and the test almost certainly catches them. But among the other 9,999 healthy people, the test's 1% false-positive rate means roughly 100 of them also test positive, purely by test error. So among everyone who tests positive — about 101 people — only 1 actually has the disease. A "99% accurate" test, applied to a rare condition, is right about 99% of the time overall, yet a positive result is correct only about 1% of the time. The test's accuracy and the probability you actually care about (given a positive, do I have it?) are two completely different numbers, and the gap between them is entirely explained by the base rate.

The fallacy, one symbol at a time

Let $D$ be the event "the thing is actually true" (has the disease, regime shift is really happening), and $+$ be the event "the signal/test came back positive." What you're told is the signal's accuracy conditional on the truth: $P(+ \mid D)$, the true-positive rate, and $P(- \mid \neg D)$, the true-negative rate. What you actually want is the reverse: $P(D \mid +)$, the probability the thing is true given a positive signal. These are not the same quantity, and confusing them is the fallacy itself.

Bayes' theorem links them, and the base rate, $P(D)$ — how common the event is before any signal — is the missing ingredient:

$$P(D \mid +) = \frac{P(+ \mid D)\, P(D)}{P(+ \mid D)\,P(D) + P(+ \mid \neg D)\,P(\neg D)} .$$

In plain English: the chance the thing is really true, given a positive signal, depends on three numbers — how often the signal correctly fires when true ($P(+\mid D)$), how often it falsely fires when not true ($P(+\mid \neg D)$), and crucially how common the thing was to begin with ($P(D)$). The denominator sums up every way a positive signal can occur — from real cases and from false alarms — and the base rate determines how much each source contributes. When $P(D)$ is tiny, the false-alarm term (which multiplies the much larger $P(\neg D) \approx 1$) can dominate the numerator even though the test rarely misfires on any individual healthy case.

Worked example 1: the disease test by the numbers

Take a population of $N = 10{,}000$. Base rate $P(D) = 0.0001$ (1 in 10,000), so 1 person truly has the disease and 9,999 don't. Test sensitivity $P(+\mid D) = 0.99$, specificity $P(-\mid \neg D)=0.99$ so $P(+\mid \neg D)=0.01$.

Expected true positives: $1 \times 0.99 = 0.99$. Expected false positives: $9{,}999 \times 0.01 \approx 99.99$. Total positives: $0.99 + 99.99 = 100.98$. So

$$P(D \mid +) = \frac{0.99}{100.98} \approx 0.0098, \quad \text{about } 1\% .$$

A "99% accurate" test yields a positive result that's correct only about 1% of the time, because the disease is 10,000 times rarer than the test's 1% error rate — with a base rate that low, false alarms from the enormous healthy population swamp the tiny number of true cases almost no matter how good the test is.

Worked example 2: the regime-shift signal by the numbers

Regime shifts occur on 1% of trading days: $P(D) = 0.01$. The signal has $P(+\mid D) = 0.95$ (fires correctly 95% of the time there's a real shift) and $P(+\mid \neg D) = 0.05$ (fires falsely on 5% of ordinary days). Out of 1,000 days: 10 are true regime-shift days, 990 are ordinary. True positives: $10 \times 0.95 = 9.5$. False positives: $990 \times 0.05 = 49.5$. Total positives: $59$. So

$$P(D \mid +) = \frac{9.5}{59} \approx 0.161, \quad \text{about } 16\% .$$

Despite a signal that sounds highly accurate (95% detection rate), when it fires there's only about a 1-in-6 chance a genuine regime shift is actually underway — the other roughly 5-in-6 firings are false alarms from the far more numerous ordinary days. Trading on this signal as if it were 95% reliable would mean reacting to false alarms five times more often than to real events.

What this means in practice

The base rate fallacy shows up constantly in quant work: evaluating anomaly detectors, fraud flags, "unusual options activity" alerts, or any classifier applied to a rare target. The single most useful habit is to always ask "how common is the thing I'm testing for?" before trusting a headline accuracy number, and to compute the full confusion-matrix breakdown (see confusion matrix and classification metrics) rather than quoting sensitivity alone. It's also why cost-sensitive thresholding matters: when false alarms vastly outnumber true positives at a rare base rate, the right response is often to raise the signal's threshold, accepting a lower detection rate in exchange for far fewer false alarms, rather than treating every firing as actionable.