How high does the bar climb when you test 100 signals?

Question

A quant screens 100 signals with two-sided z-tests, keeping the family-wise error rate at 5% via Bonferroni. For a single test the familiar critical value is $|z| = 1.96$ .

What critical $|z|$ must each of the 100 signals now clear, and what does the taller bar do to your ability to detect real signals?

Accepted Answer

Shrink the per-test level first. Bonferroni splits the 5% budget over 100 tests, so each test runs at \frac{\alpha}{m} = \frac{0.05}{100} = 0.0005 \quad (	ext{two-sided}). Turn the level into a z-hurdle. A two-sided level of 0.0005 puts 0.0005/2 = 0.00025 in each tail, so the critical value is the z with cumulative probability 1 - 0.00025 = 0.99975: z \approx 3.48. So instead of clearing 1.96, every signal must now clear about 3.48. Why this hammers power. A test's power is its chance of catching a real effect, and it depends on how far the true effect sits beyond the critical bar. Raising t…

How high does the bar climb when you test 100 signals?

Your answer

More Statistics questions