Twenty guardrails and a near-certain false alarm

Question

A mature experiment platform tracks 20 guardrail metrics on every test, each tested at $\alpha = 5\%$ . Consider a treatment that truly affects none of them.

What is the probability that at least one of the 20 guardrails falsely fires?

Accepted Answer

More metrics, more chances for a fluke. Each guardrail tested at 5% has a 95% chance of staying quiet when the treatment truly does nothing. With 20 metrics, the chance they all stay quiet shrinks fast. Compute it. Treating the guardrails as roughly independent: 1 - 0.95^{20} = 1 - 0.3585 = 0.6415 \approx 64\%. So about 64% of harmless experiments will trip at least one of the 20 guardrails by chance alone, close to two out of three. What this implies. At 20 metrics, a tripped guardrail is more likely than not even when the treatment is completely safe. Reacting to every alert would mean bloc…

Twenty guardrails and a near-certain false alarm

Your answer

More Statistics questions