Minimum Detectable Effect

Before running a test, it is worth asking the question backward: given the amount of data you're about to collect and the noise you already know is in it, what is the smallest true effect your test would even be capable of noticing? Run a test with too little data against a genuinely small edge, and the honest answer might be "you had no realistic chance of detecting this, even if it were completely real." The minimum detectable effect (MDE) is that threshold, computed in advance, and it turns "we found nothing" into a question with a real answer: was there nothing there, or was the test simply too weak to see it?

An analogy: the smallest object a telescope can resolve

Every telescope has a resolving power — the smallest angular separation between two stars it can actually distinguish, set by its aperture size and the wavelength of light. Point a small telescope at two stars closer together than its resolving power, and it will show you one blurred dot, not two, no matter how long you stare or how real the gap between them is. Asking "how much data do I need to detect an effect of size X?" and asking "given my data, what is the smallest effect X I could detect?" are the same telescope-resolving-power question asked in opposite directions.

The formula, one symbol at a time

For a two-sample comparison (say, comparing average returns under a treatment and a control), the minimum detectable effect at a chosen significance level $\alpha$ and desired power $1-\beta$ is approximately:

$$\text{MDE} = (z_{1-\alpha/2} + z_{1-\beta}) \times \sigma \sqrt{\frac{2}{n}}$$

In plain English: the smallest gap your test can reliably catch grows with how much noise is in the data ($\sigma$) and shrinks as your sample size ($n$) grows, at a rate governed by the square root of $n$. The $z$ terms are fixed constants for standard choices — at $\alpha=0.05$ two-sided and power $80\%$, $z_{1-\alpha/2} = 1.96$ and $z_{1-\beta} = 0.84$, so their sum, $2.80$, is a number worth memorizing as the standard "detectability multiplier."

Watch the running average above settle inside its shrinking envelope as more draws accumulate — that envelope is a $1/\sqrt{n}$ band, the exact same shape that shrinks the MDE. It narrows fast at first, then flattens, so doubling your sample size does not halve the smallest effect you can detect; it only shrinks it by a factor of $\sqrt{2} \approx 1.41$.

Worked example 1: MDE for a strategy A/B comparison

You plan to compare 200 trading days under Strategy A against 200 days under Strategy B ($n=200$ per group), with daily return noise $\sigma = 1.5\%$ in each group, at the standard $\alpha=0.05$, power $80\%$.

$$\text{MDE} = 2.80 \times 1.5\% \times \sqrt{\frac{2}{200}} = 2.80 \times 1.5\% \times \sqrt{0.01} = 2.80 \times 1.5\% \times 0.1$$ $$\text{MDE} = 2.80 \times 0.15\% = 0.42\%$$

With 200 days per group, the test can reliably detect a true average daily-return gap of about 0.42 percentage points or larger. If the real edge between the two strategies is smaller than that — say, a genuine but modest 0.2-point difference — this test is very unlikely to find it, regardless of how carefully it's run.

Worked example 2: what more data buys you

Quadruple the sample to $n=800$ per group, same noise:

$$\text{MDE} = 2.80 \times 1.5\% \times \sqrt{\frac{2}{800}} = 2.80 \times 1.5\% \times \sqrt{0.0025} = 2.80 \times 1.5\% \times 0.05 = 0.21\%$$

Quadrupling the data halved the MDE, from 0.42% to 0.21% — exactly the $\sqrt{n}$ relationship predicted: to detect an effect half as large, you need four times as much data, not twice. This is the concrete arithmetic behind why chasing genuinely small edges with more history has fast-diminishing returns.

What this means in practice

Before backtesting a strategy or running a live A/B trading experiment, computing the MDE tells you whether the test is even worth running for the size of edge you're hoping to find. A signal expected to add a genuine but modest 0.1% of extra return, tested against a setup whose MDE is 0.4%, is being tested by an instrument that cannot resolve it — a "no significant result" outcome would be almost guaranteed regardless of whether the signal is real, and running the test at all would waste time and risk a false conclusion of "the strategy doesn't work."