Hypothesis Testing

A hypothesis test is a decision rule under uncertainty: given noisy data, decide whether to reject a default claim. For a quant this is the machinery that separates a real edge from a lucky backtest, is this strategy's Sharpe genuinely positive, is this factor loading real, does this series have a unit root? The Neyman–Pearson framework makes the decision, and its accounting of the two ways you can be wrong is the part most people get sloppy about and desks pay for.

The setup: null, alternative, and two errors

State a null hypothesis $H_0$ (the skeptical default, "no edge," $\mu = 0$) and an alternative $H_1$. A test is a rule that maps the data to {reject, don't reject} $H_0$, defined by a rejection region. Because the data are random, two errors are possible:

	$H_0$ true	$H_1$ true
Reject $H_0$	Type I error ($\alpha$)	correct
Don't reject	correct	Type II error ($\beta$)

The type I error rate (size, significance level) is $\alpha = \mathbb{P}(\text{reject } H_0 \mid H_0)$, a false discovery. The type II error rate is $\beta = \mathbb{P}(\text{don't reject} \mid H_1)$, a missed discovery. The power is $1 - \beta = \mathbb{P}(\text{reject} \mid H_1)$, the probability of catching a real effect. The Neyman–Pearson philosophy is asymmetric: fix $\alpha$ at a tolerable level (say 5%), then find the test that maximizes power.

Constructing a test statistic

The standard recipe: form a test statistic $T$ whose distribution under $H_0$ is known, and reject when $T$ lands in the tail. For a sample mean with known variance, the $z$-statistic

$z = \frac{\bar x - \mu_0}{\sigma/\sqrt{n}} \ \xrightarrow{d}\ \mathcal{N}(0,1) \quad \text{under } H_0: \mu = \mu_0,$

by the The Central Limit Theorem. A two-sided test at level $\alpha$ rejects when $|z| > z_{1-\alpha/2}$ (e.g. $1.96$ for $\alpha = 0.05$). When $\sigma$ is estimated, $T$ follows a $t_{n-1}$ distribution. The rejection threshold is the critical value; comparing $T$ to it is mechanically identical to comparing a p-value to $\alpha$.

The Neyman–Pearson lemma

For a simple null $H_0: \theta = \theta_0$ against a simple alternative $H_1: \theta = \theta_1$, which test is most powerful at level $\alpha$? The Neyman–Pearson lemma answers definitively: the most powerful test rejects when the likelihood ratio exceeds a threshold,

$\Lambda = \frac{L(\theta_1)}{L(\theta_0)} > k,$

with $k$ chosen so the size is exactly $\alpha$. The intuition is exactly right, reject in favor of $\theta_1$ precisely where the data are relatively more probable under $\theta_1$ than under $\theta_0$, and rank observations by that ratio. Every optimal test in classical statistics is, at bottom, a likelihood-ratio comparison.

The likelihood-ratio, Wald, and score tests

Real hypotheses are usually composite (e.g. $H_0: \mu \le 0$, or nested models). The generalized likelihood-ratio test uses

$\lambda = \frac{\sup_{\theta \in \Theta_0} L(\theta)}{\sup_{\theta \in \Theta} L(\theta)},$

the ratio of the best fit under the null to the best fit overall. Wilks' theorem gives its asymptotic null distribution: $-2\log\lambda \xrightarrow{d} \chi^2_q$, where $q$ is the number of restrictions. This is the standard way to test whether adding parameters (an extra factor, a GARCH term) significantly improves fit. Two asymptotically equivalent cousins complete the "holy trinity":

• Wald test, estimate the unrestricted model and check whether the restriction is far from satisfied, scaled by the estimate's covariance: $W = (R\hat\theta - r)^\top [\operatorname{Var}]^{-1}(R\hat\theta - r)$. The ordinary $t$- and $F$-statistics are Wald tests.
• Score (Lagrange multiplier) test, estimate only the restricted model and check whether the score is far from zero there. Cheap when the restricted model is easy to fit; the standard ARCH and Breusch–Pagan tests are LM tests.

All three agree asymptotically but can differ in finite samples; Wald is sensitive to how the restriction is parameterized, LR is invariant.

Worked example: is a strategy's Sharpe positive?

A strategy returns a mean $\bar r$ with sample volatility $\hat\sigma$ over $n$ periods. Test $H_0: \mu = 0$ against $H_1: \mu > 0$. The $t$-statistic is $t = \bar r /(\hat\sigma/\sqrt n) = \widehat{\text{SR}}\cdot\sqrt{n}$, the $t$-stat is just the annualized-appropriately Sharpe times the root of the sample size. With a Sharpe of $0.5$ (annual) over 4 years, $t \approx 0.5 \times 2 = 1.0$: not close to the $1.65$ one-sided threshold. To get $t = 1.65$ at Sharpe $0.5$ you need about 11 years. This computation is why claims of "significant" edge from short backtests deserve suspicion, the power to detect realistic Sharpes is low, and the multiple-testing inflation of $\alpha$ (below) makes the false-discovery problem worse.

Failure modes in financial data

• Autocorrelation destroys the null distribution. The $t = \widehat{\text{SR}}\sqrt n$ formula assumes i.i.d. returns; serial correlation and overlapping windows make the true standard error larger, so naive $t$-stats over-reject. Use HAC (Newey–West) standard errors. See Autocorrelation and Serial Correlation.
• Low power is the norm. Real edges are small and data are short; most honest tests of trading signals are badly underpowered, so "failed to reject" almost never means "no effect."
• Non-standard distributions under the null. Testing on the boundary (a variance $= 0$, a unit root) breaks the usual $\chi^2$/normal asymptotics, Dickey–Fuller statistics have their own tables precisely for this reason. See Unit Roots and the ADF Test.
• Data snooping. Choosing the hypothesis after seeing the data, or testing thousands of signals, invalidates the fixed-$\alpha$ guarantee entirely. This is the single biggest inferential danger in quant research. See p-values and Multiple Testing and Overfitting.

In interviews

Define type I and type II errors precisely and without hesitating, swapping them is an instant tell. Explain that you fix $\alpha$ and then maximize power, and be able to state the Neyman–Pearson lemma (most powerful test is a likelihood-ratio threshold). Know the trinity, LR, Wald, score, and one distinguishing fact each (Wald needs only the unrestricted fit, score needs only the restricted fit, LR needs both and is parameterization-invariant). The killer applied question is "your backtest shows $t = 2.1$, are you convinced?" and the graduate answer names autocorrelation-inflated standard errors and multiple testing before celebrating.