With ten guardrails, how often does one falsely fire?

Question

Your experiment platform automatically checks 10 guardrail metrics on every A/B test, each with a two-sided significance test at $\alpha = 5\%$ . Suppose a particular treatment truly affects none of them (all 10 are null).

What is the probability that at least one guardrail falsely fires, and what does this imply for how you read guardrail alerts?

Accepted Answer

Each guardrail is its own coin flip. With a 5% test, a truly-null metric has a 95% chance of correctly staying quiet and a 5% chance of a false alarm. That is fine for one metric. The problem is that you are running 10 such tests at once, and each is another chance for a false alarm. Probability at least one fires. Treating the guardrails as roughly independent, the chance that all 10 correctly stay quiet is 0.95^{10}, so the chance that at least one fires is 1 - 0.95^{10} = 1 - 0.5987 = 0.4013 \approx 40\%. So even when the treatment harms nothing, about 40% of experiments will trip at least…

With ten guardrails, how often does one falsely fire?

Your answer

More Statistics questions