A/B Testing

An A/B test is the cleanest way to answer "did my change help?" You take one population, split it at random into two groups, show group A the old thing and group B the new thing, and measure some outcome, a click-through, a conversion, a fill rate, an execution cost. Randomization is the whole trick: because people land in A or B by coin flip, the two groups are on average identical in every other respect, so any difference in the outcome can be pinned on the change rather than on who happened to be in which group. In trading the same discipline shows up in transaction-cost analysis, routing order flow to two execution algorithms and comparing slippage.

The core test

For a yes/no outcome like conversion, each group gives a proportion. Call them $\hat p_A$ and $\hat p_B$ with sample sizes $n_A$ and $n_B$. The null hypothesis is that the true rates are equal; the difference $\hat p_B - \hat p_A$ is just sampling noise. To test it, pool the two groups for the shared rate $\hat p = (x_A + x_B)/(n_A + n_B)$ (where $x$ is the count of successes), compute the standard error, and form a z-statistic:

$$z = \frac{\hat p_B - \hat p_A}{\sqrt{\hat p (1-\hat p)\left(\tfrac{1}{n_A} + \tfrac{1}{n_B}\right)}} .$$

The numerator is the observed lift; the denominator is how big a lift you would expect from noise alone. If $z$ is large (past about $1.96$ for a two-sided $5\%$ test), the difference is bigger than noise can comfortably explain and you call it real.

Worked example

You test a new checkout page. Over a week:

Group	Visitors $n$	Conversions $x$	Rate $\hat p$
A (control)	1000	100	10.0%
B (variant)	1000	130	13.0%

The variant looks $3$ percentage points better, but is that real? Pool the rate: $\hat p = 230/2000 = 0.115$. The standard error is

$$\sqrt{0.115 \times 0.885 \times \left(\tfrac{1}{1000} + \tfrac{1}{1000}\right)} = \sqrt{0.115 \times 0.885 \times 0.002} \approx 0.0143 .$$

So

$$z = \frac{0.13 - 0.10}{0.0143} \approx 2.10 ,$$

which gives a two-sided p-value of about $0.036$, under $0.05$. The lift clears the bar: you would see a gap this large by chance only about $1$ time in $28$ if the pages were truly equal. Report it with a confidence interval on the difference too, roughly $3\% \pm 2.8\%$, so you know the effect could be as small as a fraction of a point.

Plan the sample size before you start

The most common failure is running a test that never had a chance of detecting the effect you cared about. The number of subjects per group needed to detect a difference $\delta$ scales like

$$n \approx \frac{(z_{\alpha/2} + z_{\beta})^2 \, \cdot\, 2\,\bar p(1-\bar p)}{\delta^2} ,$$

where $z_{\alpha/2}$ and $z_\beta$ come from your chosen significance and power. The key lesson is the $\delta^2$ in the denominator: halving the effect you want to catch quadruples the sample you need. Decide the minimum lift worth caring about, then compute $n$, before collecting a single data point.

Where A/B tests go wrong

• Peeking. Checking the result repeatedly and stopping the moment it hits significance dramatically inflates false positives, every peek is another chance for noise to cross the line. If you genuinely need to monitor as data arrives, use sequential testing, which is built for exactly that.
• Many metrics, many chances. Test one change against twenty metrics at $5\%$ each and you will "find" something. This is a multiple-testing problem; pre-register one primary metric.
• Sample ratio mismatch. If your 50/50 split arrives as 52/48, something is broken in the randomization or logging, and the whole result is suspect.
• Statistical vs practical significance. With enough traffic a trivial lift becomes "significant." Ask whether the effect is large enough to matter, not merely whether $p < 0.05$.

An A/B test is just Hypothesis Testing with randomization bolted on, and its two great enemies, peeking and running many comparisons, lead straight into Sequential Hypothesis Testing and false-discovery-rate control.