A borderline click-through test on 5000 each

Question

An ad test shows each variant to 5000 users. Variant A gets 500 clicks ( $10.0\%$ ); variant B gets 560 clicks ( $11.2\%$ ).

Compute the two-sided p-value for the difference in click rates, and decide whether B is significantly better at the 5% level.

Accepted Answer

Pooled rate under the null. \hat{p} = \frac{500 + 560}{5000 + 5000} = \frac{1060}{10000} = 0.106. Standard error. SE = \sqrt{0.106 	imes 0.894 	imes \left(	frac{1}{5000} + 	frac{1}{5000}ight)}. Now 0.106 	imes 0.894 = 0.094764, times 0.0004 = 0.0000379, and \sqrt{0.0000379} \approx 0.006157. z and p. The observed difference is 0.112 - 0.100 = 0.012: z = \frac{0.012}{0.006157} \approx 1.95, \qquad p = 2\,[1 - \Phi(1.95)] = 2 	imes 0.0256 \approx 0.051. The verdict. At p \approx 0.051, the difference is just barely not significant at the 5% level. Despite 5000 users per arm and a real-l…

A borderline click-through test on 5000 each

Your answer

Discussion

Learn the concepts

Related questions