One guardrail metric out of twenty went red

Question

Your A/B test has one primary metric plus $20$ guardrail metrics (latency, error rate, fill quality, and so on), each tested at $\alpha = 5\%$ . The primary metric is flat, but one guardrail came back significant with $p = 0.04$ , and a teammate wants to block the launch.

What is the statistical flaw in reacting to that single red metric, and what should you do instead?

Accepted Answer

The problem: multiple comparisons. Each test at \alpha = 5\% has a 5\% chance of a false positive when the true effect is zero. Run 20 independent such tests on a feature that changes nothing, and the chance that at least one trips is 1 - (1 - 0.05)^{20} = 1 - 0.95^{20} \approx 1 - 0.36 = 0.64. So even under a completely harmless feature you would expect to see roughly one red guardrail about two times out of three. A single p = 0.04 among 20 metrics is exactly what pure noise produces; it is weak evidence of real harm. What to do instead. 1. Adjust for multiplicity. Bonferroni tests each met…

One guardrail metric out of twenty went red

Your answer

More Statistics questions