Quant Memo
Advanced

High-Dimensional Mean Testing When p Exceeds n

Testing whether a vector of many means (more variables than observations) is zero requires different tools than the classical Hotelling test, which breaks down entirely once variables outnumber observations.

The classical way to test whether a vector of pp means is jointly zero — say, whether pp different factor strategies each have zero average return, tested together rather than one at a time — is Hotelling's T2T^2 test, which requires inverting the sample covariance matrix of the pp variables. That inversion needs the sample size nn to exceed pp; once you have more candidate variables than observations (thousands of factors, only a few hundred months of data), the sample covariance matrix is singular and can't be inverted at all, so the classical test simply doesn't run.

High-dimensional mean tests replace the matrix inversion with statistics that avoid it entirely — a common approach (Bai-Saranadasa and its descendants) uses a test statistic built from the sum of squared standardized means, roughly j=1pxˉj2\sum_{j=1}^p \bar{x}_j^2 adjusted for the estimated variances, rather than the full quadratic form that Hotelling's test requires. This sidesteps the singular-matrix problem because it never needs a full covariance inverse, only diagonal or low-rank pieces of it, and its distribution under the null can be derived even as pp grows alongside (or faster than) nn.

The tradeoff is power: these tests are most sensitive to many small effects spread broadly across the pp variables, and comparatively weak at detecting a signal concentrated in just a few of them — the opposite profile of a sparsity-based test like one built on Lasso selection.

When more variables than observations rules out inverting the sample covariance matrix, high-dimensional mean tests replace Hotelling's T2T^2 with statistics built from sums of squared standardized means that never require a full inverse, at the cost of losing power against signal concentrated in only a few variables.

Related concepts

Practice in interviews

Further reading

  • Bai, Saranadasa, Effect of High Dimension on a Test of Two Sample Means
ShareTwitterLinkedIn