Sequential Hypothesis Testing

A classical hypothesis test fixes the sample size up front: collect exactly $n$ observations, then decide. Sequential testing throws that out. It looks at the data as it streams in and stops the instant the evidence is strong enough to call it, one way or the other. The payoff is real: when the truth is clear, you reach a verdict with far fewer observations than a fixed-sample test would demand. Abraham Wald invented the idea during World War II to cut the number of munitions that had to be destroyed in quality-control tests, and it is exactly the right frame for anyone who wants to monitor an A/B test live.

The sequential probability ratio test

Wald's SPRT keeps a running tally of how much the data favors one hypothesis over the other. After $n$ observations, form the likelihood ratio, how probable the data is under the alternative $H_1$ divided by how probable it is under the null $H_0$:

$$\Lambda_n = \prod_{i=1}^{n} \frac{f_1(x_i)}{f_0(x_i)} .$$

Each new data point multiplies the tally by a factor bigger than one if it looks like $H_1$, smaller than one if it looks like $H_0$. You set two boundaries and follow one rule:

• if $\Lambda_n \ge A$, stop and accept $H_1$;
• if $\Lambda_n \le B$, stop and accept $H_0$;
• otherwise, keep sampling.

Wald showed the boundaries that (approximately) deliver a false-positive rate $\alpha$ and false-negative rate $\beta$ are

$$A = \frac{1-\beta}{\alpha}, \qquad B = \frac{\beta}{1-\alpha} .$$

In practice you track the logarithm of the ratio, which turns the product into a running sum, a random walk that drifts toward whichever boundary the truth lies near.

Worked example: setting the boundaries

You want a false-positive rate $\alpha = 0.05$ and false-negative rate $\beta = 0.10$ (so $90\%$ power). The boundaries are

$$A = \frac{1 - 0.10}{0.05} = \frac{0.90}{0.05} = 18, \qquad B = \frac{0.10}{1 - 0.05} = \frac{0.10}{0.95} \approx 0.105 .$$

In log terms, accept $H_1$ when the cumulative log-ratio rises above $\ln 18 \approx 2.89$, and accept $H_0$ when it falls below $\ln 0.105 \approx -2.25$. Each observation adds $\ln\!\big(f_1(x)/f_0(x)\big)$ to the tally; you simply watch for a crossing. If the truth is clearly $H_1$, the walk drifts up quickly and you often stop in a fraction of the fixed-sample size, Wald's theory shows expected savings around $50\%$ when either hypothesis is clearly true.

Why this beats peeking at a fixed-sample test

Repeatedly checking an ordinary test and stopping when it first hits $p < 0.05$ is statistically illegal, each look is another chance for noise to cross the line, and the true false-positive rate can balloon well past the nominal $5\%$. The SPRT is the honest version of the same wish: its boundaries are designed to be checked after every observation, so continuous monitoring is a feature, not a violation. The deep reason is that the likelihood ratio is a martingale under the null, and the optional stopping theorem is exactly what keeps the error rate controlled no matter when you stop.

Cautions and cousins

• You must specify the alternative. The classic SPRT tests a simple $H_0$ against a simple $H_1$; you need a concrete effect size to build the likelihood ratio. Composite versions and generalized ratios exist but are more involved.
• Random stopping time. The sample size is now itself random; you cannot promise exactly when the test ends, only its distribution.
• Modern variants. Group-sequential designs (alpha spending, O'Brien–Fleming boundaries) allow a fixed number of interim looks, and "always-valid p-values" / e-values give anytime-valid inference for streaming experiments, the direction most online-experimentation platforms have taken.

Sequential testing is the principled cure for the peeking pitfall that haunts A/B testing, it leans directly on Martingales and the Optional Stopping Theorem, and it trades a fixed sample size for a decision that arrives as soon as the data is ready to give one.