Tightening ten guardrails to a 1% threshold

Question

A team watches 10 guardrail metrics, but instead of the usual 5% they test each at a stricter $\alpha = 1\%$ . Consider a treatment that truly affects none of them.

What is the probability that at least one of the 10 guardrails falsely fires at this stricter threshold?

Accepted Answer

Stricter tests fire less often. At a 1% threshold, a harmless metric stays correctly quiet 99% of the time. With 10 metrics, the chance all stay quiet is 0.99^{10}. Compute it. 1 - 0.99^{10} = 1 - 0.9044 = 0.0956 \approx 10\%. So about 10% of harmless experiments trip at least one guardrail, down sharply from the 40% you would get with the same 10 metrics at a 5% threshold. What this shows. Tightening the per-metric threshold is exactly how you keep the overall false-alarm rate under control when you watch many metrics. This is the logic behind a Bonferroni correction: to get the overall fals…

Tightening ten guardrails to a 1% threshold

Your answer

Discussion

Learn the concepts

Related questions