Statistical Power and Sample Size

You run a test, the p-value comes back at 0.31, and you write in your notes: "no effect". But two very different worlds produce that result. In one, there genuinely is nothing there. In the other, there is something worth millions and your test simply could not see it. Nothing in the p-value distinguishes those worlds. The quantity that does is power, and it is the single most neglected number in applied statistics.

The blurry camera

Imagine photographing a dark room to check whether anyone is standing in the corner. A cheap phone camera returns a grainy mess. You look at it and conclude "the room is empty." That conclusion is worthless — not because the photo is wrong, but because that camera could not have shown a person even if one were there. A better lens, a longer exposure, more light: any of those raise your chance of spotting the figure if the figure exists.

Power is exactly that: the probability your study would detect the effect, assuming the effect is real and of a stated size. A study with 30% power is the grainy phone camera. Running it and finding nothing tells you almost nothing about the room.

The four numbers that move together

Every power calculation juggles four quantities, and fixing any three pins down the fourth.

• The false-alarm rate $\alpha$ — the chance you shout "effect!" when there is none. Conventionally 5%.
• The miss rate $\beta$ — the chance you shrug "nothing here" when there really is something. Power is $1 - \beta$, and 80% is the usual floor.
• The effect size $\delta$ — the Greek letter delta, how big the real difference is, in the same units you measure.
• The noise $\sigma$ — sigma, the standard deviation of a single observation, and $n$, the number of observations.

The picture below shows why these four are locked together. Two bell curves: the left one is what your test statistic looks like when nothing is going on, the right one is what it looks like when the effect is real. You must draw a line somewhere and call everything to its right "significant."

Notice what the picture forbids. With the two curves where they are, tightening $\alpha$ pushes the cutoff right and shrinks power. The only way to win on both sides is to make the curves narrower, and the curves narrow as the square root of sample size. That is the whole lever.

The formula, and the number to remember

For comparing two groups of size $n$ each, the requirement is

$$n = \frac{2\,(z_{\alpha/2} + z_{\beta})^2\,\sigma^2}{\delta^2} .$$

In words: the sample you need grows with the square of the noise-to-effect ratio, multiplied by a constant that depends only on how strict and how sensitive you insist on being. The two $z$ terms are the standard-normal cutoffs for your chosen error rates: for the usual 5% two-sided test, $z_{\alpha/2} = 1.96$; for 80% power, $z_{\beta} = 0.84$.

Add them and you get 2.80, and $2 \times 2.80^2 = 15.7$, which rounds to a number worth carrying in your head:

$$n \approx \frac{16\,\sigma^2}{\delta^2} \quad \text{per group.}$$

In words: at the standard settings, you need about sixteen times the squared ratio of noise to effect, in each arm. Halve the effect you want to detect and the sample quadruples. This is why detecting small edges is so brutally expensive.

Push the sample size slider up in the explorer above and watch the histogram of sample means pull in tight around the truth. That narrowing is power. Nothing else in the picture changed — same underlying world, same effect — you simply gained the resolution to see it.

Worked example one: how long to trust a Sharpe ratio?

A strategy claims a Sharpe ratio of 0.5. How many years of returns do you need before you could distinguish it from zero?

Step 1 — write the test statistic. Over $T$ years, the t-statistic of a strategy's mean return is approximately $\text{SR} \times \sqrt{T}$, because the Sharpe ratio is already a mean-to-noise ratio per year.

Step 2 — set the bar. For 5% two-sided significance and 80% power we need that statistic to reach $1.96 + 0.84 = 2.80$.

Step 3 — solve.

$$0.5\sqrt{T} \ge 2.80 \;\Longrightarrow\; \sqrt{T} \ge 5.6 \;\Longrightarrow\; T \ge 31.4 .$$

Thirty-one years. A strategy with a genuine Sharpe of 0.5 needs three decades of live returns before a standard test reliably notices it. Repeat the sum for other Sharpes:

True Sharpe	Years needed
0.3	87
0.5	31
1.0	8
2.0	2

This one calculation explains a great deal about the industry. Almost no one has thirty years of clean data on a single strategy, so almost no moderate-Sharpe claim has ever been properly tested. It also explains why backtests are seductive: it is far easier to find a lucky 5-year window than to earn a real edge.

Worked example two: sizing an execution A/B test

Two order-routing algorithms. The incumbent fills 60% of passive orders; you want to detect an improvement to 63%.

Step 1 — effect and noise. The effect is $\delta = 0.63 - 0.60 = 0.03$. For a yes/no outcome the variance of one observation is $p(1-p) = 0.60 \times 0.40 = 0.24$.

Step 2 — plug in.

$$n = \frac{2 \times (2.80)^2 \times 0.24}{(0.03)^2} = \frac{2 \times 7.84 \times 0.24}{0.0009} = \frac{3.763}{0.0009} = 4181 .$$

Step 3 — read it. About 4,182 orders through each router, roughly 8,400 in total. If you route 500 passive orders a day, that is about seventeen trading days — genuinely feasible, so run it properly.

Step 4 — feel the square law. Chasing a 1.5-point improvement instead of 3 points needs $4181 \times 4 = 16{,}724$ orders per arm, about ten weeks. Chasing 0.75 points needs nearly a year.

The flattening matters. Once you are past 80%, extra data buys very little detection and a lot of calendar time — which is why 80% became the convention rather than 95%.

What this means in practice

The first use is triage. Before running any study, ask what effect would change your decision, then compute the sample it needs. If the answer is "more data than exists," you have learned something valuable for free: this question is not answerable, and you should redesign rather than run it and squint at the output.

The second use is reading other people's results. An underpowered study that does hit significance is not reassuring — it is alarming. To clear the bar with too little data, the observed effect must be freakishly large, so the published estimate is systematically inflated. That is a major reason headline effects shrink on replication and backtested Sharpes decay after launch; see The Deflated Sharpe Ratio.

The third use is defensive. When a colleague reports "we tested it, no effect," the follow-up is not "are you sure?" but "what could you have detected?" If the honest answer is "a 40% improvement," their null result rules out nothing you cared about.

Once you can size a study you can size a signal, and the same arithmetic underpins Minimum Detectable Effect, A/B Testing design, and honest Confidence Intervals.