Effect Size and Cohen's d

Two backtests both report "statistically significant, $p < 0.01$." One found a strategy that beats its benchmark by 0.02% a year; the other found one that beats it by 8% a year. Significance alone cannot tell them apart, with enough data, even a genuinely tiny, economically meaningless difference clears a $p$-value threshold. What you actually want to know is how big the difference is, measured on a scale you can reason about. That is effect size, and Cohen's d is its most common form.

An analogy: grading on a curve versus grading in raw points

Saying "Strategy A beat Strategy B by 3 percentage points a year" sounds informative until you ask: 3 points relative to what? If year-to-year noise in that market typically swings 30 points, a 3-point edge is a rounding error lost in the static. If typical noise is only 1 point, a 3-point edge is enormous and rare. Cohen's d is the answer to "relative to what": it reports a gap not in raw units, but in units of the data's own natural noise, like grading on a curve instead of reporting a raw score, so a gap of "0.3" means the same thing whether you're comparing test scores, drug doses, or trading strategies.

The formula, one symbol at a time

Let $\bar{x}_1$ and $\bar{x}_2$ be the two group means being compared (for instance, average returns under two strategies), and let $s_p$ be the pooled standard deviation, a blended measure of how noisy the data is within each group. Cohen's d is:

In plain English: take the raw gap between two averages and divide it by how much the individual data points naturally bounce around within each group. A $d$ of 1.0 means the two group means are a full standard deviation apart, a large, visually obvious gap. Cohen's own rough guideposts, still widely used as a starting intuition: $d \approx 0.2$ is a "small" effect, $d \approx 0.5$ "medium," $d \approx 0.8$ "large", though what counts as practically meaningful always depends on the specific field and stakes.

Picture two bell curves like the one above, shifted apart along the horizontal axis. Cohen's d is exactly how many standard-deviation widths that shift spans, a small $d$ means the two curves mostly overlap; a large $d$ means they barely touch.

Worked example 1: computing d for two strategies

Strategy A's monthly returns average 1.2% with standard deviation 3.0%; Strategy B's average 0.9% with standard deviation 2.8%, both estimated from large enough samples that the two standard deviations can be pooled by a simple average: $s_p = (3.0+2.8)/2 = 2.9$.

A $d$ of about 0.10 is a small effect by Cohen's guideposts, the two return distributions overlap heavily despite the 0.3-point difference in their averages, because month-to-month noise (2.8–3.0%) dwarfs the gap between them.

Worked example 2: same raw gap, different noise, different d

Now compare Strategy C, averaging 1.2%, against Strategy D, averaging 0.9%, the identical 0.3-point raw gap as above, but both are much steadier, with standard deviations of only 0.4% each, so $s_p = 0.4$.

The exact same raw 0.3-point gap now registers as a large effect, because the underlying noise is so much smaller. This is the entire point of standardizing: raw gaps are meaningless without knowing the scale of the noise they're competing against, and Cohen's d bakes that comparison in automatically.

What this means in practice

Reporting only a $p$-value from a strategy backtest invites exactly the confusion the two worked examples show: with a long enough history, even a strategy whose true edge is a rounding error can produce a small $p$-value, simply because the sample is huge. Reporting Cohen's d alongside it forces the honest question, is this edge actually large relative to the strategy's own noise, or is it statistically detectable but practically irrelevant? This is also the standard input to power calculations: before running a test, researchers use an assumed effect size to work out how much data they will need to reliably detect it at all.