Effect Size and Cohen's d

Two backtests both report "statistically significant, $p < 0.01$." One found a strategy that beats its benchmark by 0.02% a year; the other found one that beats it by 8% a year. Significance alone cannot tell them apart — with enough data, even a genuinely tiny, economically meaningless difference clears a $p$-value threshold. What you actually want to know is how big the difference is, measured on a scale you can reason about. That is effect size, and Cohen's d is its most common form.

An analogy: grading on a curve versus grading in raw points

Saying "Strategy A beat Strategy B by 3 percentage points a year" sounds informative until you ask: 3 points relative to what? If year-to-year noise in that market typically swings 30 points, a 3-point edge is a rounding error lost in the static. If typical noise is only 1 point, a 3-point edge is enormous and rare. Cohen's d is the answer to "relative to what": it reports a gap not in raw units, but in units of the data's own natural noise — like grading on a curve instead of reporting a raw score, so a gap of "0.3" means the same thing whether you're comparing test scores, drug doses, or trading strategies.

The formula, one symbol at a time

Let $\bar{x}_1$ and $\bar{x}_2$ be the two group means being compared (for instance, average returns under two strategies), and let $s_p$ be the pooled standard deviation — a blended measure of how noisy the data is within each group. Cohen's d is:

$$d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}$$

In plain English: take the raw gap between two averages and divide it by how much the individual data points naturally bounce around within each group. A $d$ of 1.0 means the two group means are a full standard deviation apart — a large, visually obvious gap. Cohen's own rough guideposts, still widely used as a starting intuition: $d \approx 0.2$ is a "small" effect, $d \approx 0.5$ "medium," $d \approx 0.8$ "large" — though what counts as practically meaningful always depends on the specific field and stakes.

Picture two bell curves like the one above, shifted apart along the horizontal axis. Cohen's d is exactly how many standard-deviation widths that shift spans — a small $d$ means the two curves mostly overlap; a large $d$ means they barely touch.

Worked example 1: computing d for two strategies

Strategy A's monthly returns average 1.2% with standard deviation 3.0%; Strategy B's average 0.9% with standard deviation 2.8%, both estimated from large enough samples that the two standard deviations can be pooled by a simple average: $s_p = (3.0+2.8)/2 = 2.9$.

$$d = \frac{1.2 - 0.9}{2.9} = \frac{0.3}{2.9} = 0.103$$

A $d$ of about 0.10 is a small effect by Cohen's guideposts — the two return distributions overlap heavily despite the 0.3-point difference in their averages, because month-to-month noise (2.8–3.0%) dwarfs the gap between them.

Worked example 2: same raw gap, different noise, different d

Now compare Strategy C, averaging 1.2%, against Strategy D, averaging 0.9% — the identical 0.3-point raw gap as above — but both are much steadier, with standard deviations of only 0.4% each, so $s_p = 0.4$.

$$d = \frac{1.2-0.9}{0.4} = \frac{0.3}{0.4} = 0.75$$

The exact same raw 0.3-point gap now registers as a large effect, because the underlying noise is so much smaller. This is the entire point of standardizing: raw gaps are meaningless without knowing the scale of the noise they're competing against, and Cohen's d bakes that comparison in automatically.

What this means in practice

Reporting only a $p$-value from a strategy backtest invites exactly the confusion the two worked examples show: with a long enough history, even a strategy whose true edge is a rounding error can produce a small $p$-value, simply because the sample is huge. Reporting Cohen's d alongside it forces the honest question — is this edge actually large relative to the strategy's own noise, or is it statistically detectable but practically irrelevant? This is also the standard input to power calculations: before running a test, researchers use an assumed effect size to work out how much data they will need to reliably detect it at all.