Five guardrails and a one-in-four false alarm

Question

A team keeps a lean set of just 5 guardrail metrics, each tested at $\alpha = 5\%$ . Consider a treatment that truly affects none of them.

What is the probability that at least one of the 5 guardrails falsely fires?

Accepted Answer

Even a few tests add up. Each guardrail at 5% stays correctly quiet 95% of the time under a harmless treatment. With 5 metrics, the chance all stay quiet is 0.95^5. Compute it. Treating them as roughly independent: 1 - 0.95^{5} = 1 - 0.7738 = 0.2262 \approx 23\%. So even with a lean set of 5 guardrails, about 23% of harmless experiments will trip at least one by chance, roughly one in four. What this implies. Keeping the guardrail set small is itself a good way to limit false alarms: 5 metrics give 23%, versus 40% for 10 and 64% for 20. But 23% is still far from negligible, so a single trippe…

Five guardrails and a one-in-four false alarm

Your answer

Discussion

Learn the concepts

Related questions