Ten metrics on the dashboard, one lit up green

Question

An experiment dashboard shows 10 metrics. The treatment moved nothing meaningfully, but one metric came back at $p = 0.03$ . The team wants to declare a win on that metric.

Quantify the false-positive risk from scanning 10 metrics, and describe the disciplined way to handle a multi-metric readout.

Accepted Answer

The multiplicity. Even if the treatment truly does nothing, each metric independently has a 5% chance of a false positive. Across 10 metrics the expected number of spurious "hits" is 10 	imes 0.05 = 0.5 per experiment, and the probability of at least one false positive is 1 - 0.95^{10} \approx 40\% (less if the metrics are correlated, but still substantial). So finding one significant metric out of ten, when nothing is real, is completely unremarkable, you'd expect it roughly every other experiment. The single green cell is almost certainly the noise winner of a 10-way scan, and its face-val…

Ten metrics on the dashboard, one lit up green

Your answer

More Statistics questions