How Much Data You Need to Detect an Improvement

A team retrains a fraud model and wants to know if it genuinely beats the current one before switching production traffic over. They run a shadow test for two weeks, see the new model's accuracy is half a percentage point higher, and can't tell if that's a real improvement or noise from having only a few hundred fraud cases in two weeks of data. This is a sample-size problem, and it's solvable before running the test: given how big an improvement you actually care about detecting, how much data do you need to reliably tell it apart from chance?

The logic of a power calculation

Detecting a difference between two proportions (like two error rates) reliably requires enough observations that the expected gap between them is large compared to the noise in estimating each one. A standard approximate formula for the sample size needed per group to detect a difference in proportions $p_1$ and $p_2$ at a given significance level and power is:

$$n \approx \frac{2\,\bar{p}(1-\bar{p})\,(z_{\alpha/2} + z_{\beta})^2}{(p_1 - p_2)^2}$$

where $\bar{p}$ is the average of the two rates, and $z_{\alpha/2}$, $z_{\beta}$ are standard-normal cutoffs for the desired significance level and power (commonly $z_{\alpha/2}\approx 1.96$ for 95% confidence and $z_{\beta}\approx 0.84$ for 80% power). In plain English: the required sample size grows roughly with the inverse square of the improvement you're trying to detect — halving the effect size you want to detect roughly quadruples the data you need, which is why detecting small improvements reliably is disproportionately expensive.

Worked example: why two weeks wasn't enough

Suppose the current fraud model catches fraud at rate $p_1 = 0.60$ (of true fraud cases) and the new model is hoped to catch $p_2 = 0.65$ — a 5-point improvement. With $\bar p = 0.625$, $z_{\alpha/2}=1.96$, $z_\beta=0.84$:

$$n \approx \frac{2(0.625)(0.375)(1.96+0.84)^2}{(0.05)^2} = \frac{0.469 \times 7.84}{0.0025} \approx 1{,}470$$

That's roughly 1,470 fraud cases needed in each group to reliably detect a 5-point improvement — not 1,470 transactions. If fraud occurs in 1 out of 2,000 transactions, that means needing on the order of 3 million transactions per group, which could take many months, not two weeks. This is exactly why the team's two-week shadow test, with only a few hundred fraud cases total, had almost no chance of distinguishing a genuine 5-point improvement from noise — the test wasn't wrong, it was simply underpowered from the start.

The explorer above shows how the spread of a sample statistic narrows as sample size grows — the same shrinking-noise logic is what a power calculation is quantifying in reverse: how much data is needed to shrink the noise below the size of the effect you're hoping to see.

What this means in practice

Before launching a shadow deployment or an A/B test comparing models, run the power calculation using the smallest improvement that would actually matter for the business (not the improvement you hope for) and the base rate of the event of interest. If the required sample size implies a test duration that's impractical, either accept lower power going in (and interpret a "no difference" result cautiously) or find a way to increase the event rate observed, such as testing on a riskier subpopulation where positives are more common.