One significant, one not, so A beats B?

Question

Two strategies are evaluated. Strategy A has an estimated edge of $2.5$ bp with standard error $1.0$ bp (so $z_A = 2.5$ , $p \approx 0.012$ , significant). Strategy B has an estimated edge of $1.8$ bp with standard error $1.2$ bp (so $z_B = 1.5$ , $p \approx 0.13$ , not significant). A manager concludes, "A is significant and B is not, so A is meaningfully better than B."

Test whether A's edge is significantly greater than B's, and explain the fallacy in the manager's reasoning.

Accepted Answer

The fallacy. "A is significant, B is not" compares each edge to zero separately. It does not compare A to B. The difference between "significant" and "not significant" is not itself a test of whether the two edges differ. To claim A beats B, you must test the difference A - B directly. Test the difference. The estimated difference is 2.5 - 1.8 = 0.7 bp. Assuming the two estimates are independent, the standard error of the difference combines the two variances: SE{	ext{diff}} = \sqrt{SEA^2 + SEB^2} = \sqrt{1.0^2 + 1.2^2} = \sqrt{2.44} \approx 1.56. So the z-statistic for the difference is z =…

One significant, one not, so A beats B?

Your answer

Discussion

Learn the concepts

Related questions