Forty guardrails: false alarms almost guaranteed

Question

An organization has accumulated a sprawling dashboard of 40 guardrail metrics, each tested at $\alpha = 5\%$ on every experiment. Consider a treatment that truly affects none of them.

What is the probability that at least one of the 40 guardrails falsely fires?

Accepted Answer

Forty chances at a fluke. Each guardrail at 5% stays correctly quiet 95% of the time when the treatment is harmless. With 40 metrics, the chance all 40 stay quiet is 0.95^{40}, which is small. Compute it. Note 0.95^{40} = (0.95^{20})^2 = 0.3585^2 \approx 0.1285, so 1 - 0.95^{40} = 1 - 0.1285 = 0.8715 \approx 87\%. So about 87% of completely harmless experiments will trip at least one guardrail by chance, nearly guaranteed. What this shows. Piling on guardrails without correction destroys their usefulness: at 40 metrics, an alarm carries almost no information because you get one on nearly ever…

Forty guardrails: false alarms almost guaranteed

Your answer

More Statistics questions