The Law of Small Numbers

A trader tests a new signal on the last 15 trades and it wins 11 of them — a 73% hit rate — and immediately wants to scale it up before "the edge disappears." A more cautious colleague points out that 15 trades is barely enough to distinguish a genuinely 70%-accurate signal from a coin flip that got lucky. The trader isn't being irrational in an obvious way — the mistake is subtler and almost universal: people intuitively treat small samples as if they were miniature, reliable copies of the true underlying distribution, when in reality small samples are dramatically noisier than that intuition suggests. This mistaken intuition is what Tversky and Kahneman named the "law of small numbers" — a sly reference to the true law of large numbers, which small samples do not yet obey.

An analogy: judging a coin from four flips

If a fair coin is flipped four times, getting three heads and one tail (75% heads) feels only mildly surprising — most people wouldn't conclude the coin is biased. But if the same 75%-heads ratio showed up over 400 flips, everyone would correctly conclude the coin is loaded, because that ratio is now statistically almost impossible for a fair coin. The problem is that people's gut feeling of "how surprising is 75%" barely changes between 4 flips and 400 flips, even though the actual statistical evidence changes enormously. The law of small numbers is this failure to intuitively scale down your confidence when your sample is small — treating a noisy four-flip sample with nearly the same trust as a stable four-hundred-flip sample.

The statistics, one piece at a time

For a sample of $n$ independent draws from a distribution with true success probability $p$, the standard deviation of the observed sample proportion $\hat p$ is

$$\text{SD}(\hat p) = \sqrt{\frac{p(1-p)}{n}}.$$

In plain English: the noisiness of your observed win rate shrinks with the square root of the sample size, not the sample size itself — quadrupling your sample only halves the noise. This means small samples carry disproportionately large amounts of noise relative to how "informative" they feel: a human brain evolved to spot patterns quickly tends to treat $n=15$ as already representative, but $\sqrt{15} \approx 3.9$ tells you the standard deviation of your win-rate estimate is still large relative to typical trading edges, so a run of good luck is nowhere near ruled out.

Worked example 1: how noisy is 11 wins out of 15?

If the true win probability is actually $p = 0.5$ (no edge at all), the standard deviation of the observed proportion over $n=15$ trades is $\sqrt{0.5 \times 0.5 / 15} = \sqrt{0.0167} \approx 0.129$, or about 12.9 percentage points. Observing $\hat p = 11/15 \approx 0.733$ is $(0.733 - 0.5)/0.129 \approx 1.8$ standard deviations above 0.5 — noticeable, but far from the "statistically certain edge" the raw 73% figure implies, and well within the range a coin flip produces by chance more than 5% of the time in a two-sided test. The trader's instinct to trust the 73% number as if it were a stable long-run rate is precisely the law of small numbers at work.

Worked example 2: how many trades would actually confirm it?

Suppose the trader wants a win rate estimate precise enough that a true 55% edge (a very good, realistic edge in most markets) can be reliably distinguished from a coin flip, wanting the standard deviation of $\hat p$ under $p=0.5$ to be no more than 0.02 (2 percentage points) so a 5-point gap is many standard deviations away. Solving $\sqrt{0.25/n} = 0.02$ gives $n = 0.25/0.0004 = 625$ trades. Going from a "convincing-feeling" 15-trade sample to a statistically meaningful one requires over 40 times as many trades — a gap that gut instinct almost never anticipates, because the feeling of confidence from watching 11 out of 15 wins accumulate doesn't scale the way the actual required sample size does.

Watch the running average above jitter wildly over its first handful of draws before settling near the true mean — that early, noisy stretch is exactly what a trader is looking at when they judge a strategy off its first 15 trades.

What this means in practice

The law of small numbers is behind a huge share of false signals in trading: over-trusting a short backtest window, abandoning a strategy after a short losing streak that's well within normal variance, or being swayed by a small pilot sample of A/B test results. It's also why reputable quant research insists on out-of-sample testing over long, varied periods rather than a handful of favorable months — the human tendency is to treat a compelling-looking short sample as already conclusive, when statistically it's often barely more informative than noise.