Proportion Tests and Binomial Inference

A strategy wins 54 out of 100 trades. Is that a real edge, or exactly the kind of wobble you'd expect from flipping a fair coin 100 times? A fair coin flipped 100 times does not land on exactly 50 heads every time — it lands anywhere in a range around 50 purely from randomness, and 54 is well within that range. A proportion test is the tool that turns "does 54% feel like an edge" into a precise number: how surprising is 54 wins out of 100, if the true win rate were actually 50%?

An analogy: is this coin actually fair?

Suspect a coin might be weighted, and you flip it 100 times, getting 58 heads. A perfectly fair coin would not always give exactly 50 heads — sometimes 47, sometimes 53, purely by chance — so the question isn't "is 58 exactly 50?" (it obviously isn't) but "is 58 far enough from 50 that pure chance is an unconvincing explanation?" A proportion test answers exactly this: it computes how much a fair coin's result naturally wobbles around 50, and checks whether your observed count sits comfortably inside that natural wobble or clearly outside it.

The formula, one symbol at a time

Let $\hat{p}$ be the observed proportion of successes (wins, hits) out of $n$ trials, and let $p_0$ be the hypothesized true proportion under the null (often $0.5$, or a strategy's historical baseline). By the The Central Limit Theorem, for large enough $n$, the sampling distribution of $\hat p$ is approximately normal, with standard error $\sqrt{p_0(1-p_0)/n}$. The test statistic is:

$$z = \frac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$$

In plain English: take the gap between what you observed and what pure chance would predict, and divide by how much that gap naturally wobbles from sample to sample. The result, $z$, tells you how many "typical wobbles" away from the null your observation sits. A $z$ beyond roughly $\pm 1.96$ corresponds to a two-sided $p$-value below $0.05$ — the conventional threshold for calling the gap surprising enough to doubt the null.

The distribution above shows exactly the natural spread of outcomes a fair ($p=0.5$) process produces over 100 trials — most of its mass sits near 50, but 54 or even 58 successes are visibly still well within the bulk of ordinary outcomes, not out in the sparse tails.

Worked example 1: a 54-out-of-100 win rate

$\hat{p} = 54/100 = 0.54$, testing against $p_0 = 0.5$, $n=100$.

$$z = \frac{0.54 - 0.50}{\sqrt{0.5(0.5)/100}} = \frac{0.04}{\sqrt{0.0025}} = \frac{0.04}{0.05} = 0.80$$

A $z$-score of $0.80$ is well within the ordinary range of outcomes a fair coin would produce — the two-sided $p$-value is about $0.42$, nowhere near the conventional $0.05$ threshold. A 54% win rate over 100 trades is, statistically, indistinguishable from noise around a 50% baseline.

Worked example 2: the same win rate, ten times the trades

Now suppose the same 54% win rate holds up over $n = 1{,}000$ trades: $\hat{p} = 540/1000 = 0.54$.

$$z = \frac{0.54-0.50}{\sqrt{0.5(0.5)/1000}} = \frac{0.04}{\sqrt{0.00025}} = \frac{0.04}{0.0158} = 2.53$$

A $z$ of $2.53$ corresponds to a two-sided $p$-value of about $0.011$ — now comfortably below $0.05$. The identical 54% win rate flips from "unremarkable noise" to "statistically significant" purely because it held up over ten times more trades — the standard error shrank with $\sqrt{n}$, and the same signal now stands out clearly above it.

What this means in practice

Any binary, win/loss trading metric — hit rate on a directional call, fill rate on a passive order, click-through rate on a trading UI experiment — is naturally analyzed with a proportion test. It is also the standard tool for checking a strategy's realized win rate against a theoretically implied one (for example, checking whether an options strategy's assignment frequency matches its priced-in probability of exercise). The core lesson from the two worked examples generalizes: a headline win rate is meaningless without its sample size attached, because the same percentage can be either noise or signal depending entirely on how many trials produced it.