The Prosecutor's Fallacy

A risk model flags a trader's activity as matching a rare fraud pattern that occurs in only 1 in 10,000 innocent traders. The compliance team reasons: "there's only a 1-in-10,000 chance of seeing this pattern if the trader is innocent, so there's a 99.99% chance the trader is guilty." That sounds airtight, and it's exactly the reasoning that gives the prosecutor's fallacy its name — because it's the mistake real prosecutors have made in real trials. The number 1-in-10,000 answers the question "how likely is this evidence, given innocence?" It does not answer the question everyone actually cares about: "how likely is innocence, given this evidence?" Those are different numbers, and confusing them can turn an innocent person's rare-but-real coincidence into "near-certain guilt."

An analogy: tall people and basketball

Suppose only 1 in 10,000 people is over 7 feet tall. If you learn someone is over 7 feet tall, does that mean there's a 99.99% chance they're in the NBA? Obviously not — most 7-footers are not NBA players. The rarity of the trait (being 7 feet tall) among the general population says almost nothing on its own about the probability of a particular explanation for it (being a pro player), because that depends on how many other 7-footers exist and how many of them aren't NBA players. You need to know the relative sizes of both the "guilty" and "innocent" populations, not just how rare the trait is in one of them.

The logic, one piece at a time

Let $E$ be the observed evidence (the flagged pattern), and let $H$ be the hypothesis of innocence. The evidence's rarity under innocence is $P(E \mid H) = 1/10{,}000$ — a small number. The prosecutor's fallacy treats this as if it were $P(H \mid E)$, the probability of innocence given the evidence. Bayes' rule shows why these differ:

$$P(H \mid E) = \frac{P(E \mid H) \, P(H)}{P(E)}.$$

In plain English: the probability of innocence given the evidence depends not just on how rare the evidence is among innocent people, $P(E\mid H)$, but also on the base rate of innocence in the first place, $P(H)$, and on how common the evidence is overall, $P(E)$ — which includes how often guilty traders also show this pattern. If innocent traders vastly outnumber guilty ones (as they almost always do), even a rare-per-innocent-trader pattern can still be seen mostly among innocent traders in absolute terms, simply because there are so many more of them to draw from.

Worked example 1: the fraud flag, with numbers

Suppose 1,000,000 traders are innocent and only 100 are actually running the fraud scheme. Among innocent traders, the flagged pattern occurs at rate $1/10{,}000$, so $1{,}000{,}000 / 10{,}000 = 100$ innocent traders get flagged. Among the 100 guilty traders, suppose the pattern is much more common — it occurs 90% of the time — so $90$ guilty traders get flagged. Total flagged: $100 + 90 = 190$. Given a flag, the probability of actual guilt is

$$P(\text{guilty} \mid \text{flagged}) = \frac{90}{190} \approx 47\%,$$

nowhere near the naive "99.99% guilty" the raw 1-in-10,000 rarity implied. Roughly half of all flagged traders are innocent, purely because there are so many more innocent traders overall that even a rare false-positive rate produces a large absolute number of false flags.

Worked example 2: how the base rate flips the conclusion

Now suppose fraud is even rarer — only 10 guilty traders exist among the same 1,000,000 innocent ones, with the same detection rates (1-in-10,000 false-positive rate, 90% true-positive rate). Flagged innocents: still $100$. Flagged guilty: $10 \times 0.9 = 9$. Then $P(\text{guilty} \mid \text{flagged}) = 9/(100+9) \approx 8.3\%$ — the exact same "1-in-10,000 evidence" now corresponds to over 90% probability of innocence, simply because the pool of people it could have falsely flagged is so much larger than the pool it could have correctly flagged. The evidence's rarity never changed; the conclusion flipped entirely because of the base rate.

Set a low prior and feed in "evidence" above: notice how much the posterior depends on where the prior started, not just on how strong any single piece of evidence looks — the same logic that makes fraud-flag rarity alone an unreliable guide to guilt.

What this means in practice

The prosecutor's fallacy is everywhere in quant risk and compliance work: a model that flags "1-in-a-million unlikely under the null" trading patterns as fraud, without accounting for how many total traders were tested and how rare actual fraud is among them; a backtest p-value treated as "probability the strategy has no real edge," when it's really the probability of the data given no edge. Any alert system built on "how rare is this under the innocent/null scenario" needs the base rate of the thing it's trying to detect before its raw hit rate can be turned into a trustworthy probability.