Diagnosing Cross-Sectional Signal Regressions

Imagine testing whether a new diet works by measuring weight loss across many people on the same day. If those people are all members of the same household, eating the same meals, sharing the same schedule, their outcomes aren't 20 independent pieces of evidence — they're one correlated household outcome dressed up to look like 20. A regression that treats each person as an independent observation will report far more statistical confidence than the data actually supports. This is precisely the trap a cross-sectional regression of stock returns on a candidate alpha signal falls into by default: on any given date, hundreds of stocks' returns are driven by shared market and sector moves, so they are not independent observations of "does the signal work," and the same stock's returns across different dates are autocorrelated too. Diagnosing a cross-sectional signal regression means fixing exactly these two problems before trusting the headline t-statistic.

The problem, one symbol at a time

A cross-sectional signal regression, run separately at each date $t$, fits:

$$r_{i,t+1} = \alpha_t + \beta_t \, s_{i,t} + \varepsilon_{i,t+1}$$

In words: forward return of stock $i$ regressed on today's signal value $s_{i,t}$ for that stock, giving a slope $\beta_t$ each period. Ordinary OLS standard errors on a pooled version of this regression (stacking every stock-date together) assume every $\varepsilon_{i,t+1}$ is independent of every other. That assumption fails along two axes at once: cross-sectionally, residuals on the same date $t$ are correlated (shared market/sector shocks hit many stocks together), and serially, residuals on the same stock $i$ across different dates are correlated (persistence in mispricing, earnings cycles). The standard fix is the Fama-MacBeth procedure: run the cross-sectional regression separately at each date to get a time series of $\hat\beta_t$ estimates, then test whether their average is different from zero using the standard error of that time series:

$$\bar\beta = \frac{1}{T}\sum_{t=1}^T \hat\beta_t, \qquad SE(\bar\beta) = \frac{s_{\hat\beta}}{\sqrt T}$$

In words: instead of pretending you have (number of stocks) x (number of dates) independent observations, you're honestly counting your independent evidence as $T$ — the number of dates — because that's the axis along which the underlying draws are plausibly close to independent, and computing the standard error of the average slope the ordinary way across that much smaller, more honest sample size.

Drag the points in the explorer above and watch how a single point far from the cluster pulls the fitted line and widens the residual spread — that's the same mechanical vulnerability a cross-sectional signal regression has to a handful of extreme observations, worked through numerically below.

Worked example 1: why the naive t-stat overstates confidence

Six stocks on one date, with signal $s$ and forward return $r$ (in percent): $(1, 0.8), (2, 1.5), (0.5, 0.3), (1.5, 1.1), (-1, -0.9), (-0.5, -0.2)$. A quick OLS fit gives a positive slope with an apparent t-stat that, treating all 6 points as independent, might come out around $t\approx4$ — seemingly comfortable significance. But suppose these 6 stocks are all in the same sector, and an estimated intra-date (same-day) correlation of residuals of $\bar\rho = 0.4$ applies. A standard adjustment for the effective sample size under this kind of clustering shrinks the effective number of independent observations from $n=6$ toward roughly $n_{\text{eff}} = n / (1 + (n-1)\bar\rho) = 6/(1+5\times0.4) = 6/3 = 2$ — only 2 effectively-independent data points' worth of information. Standard errors computed off $n_{\text{eff}}=2$ rather than $n=6$ are far wider, and the same slope that looked like $t\approx4$ against a naive $n=6$ can easily fall to $t<2$ once the effective sample size correctly reflects the shared, correlated shock hitting all six stocks that day — no longer clearly significant.

Worked example 2: one influential observation swings the slope

Take five ordinary stocks with signal/return pairs clustered tightly around a modest positive relationship: $(0.2, 0.3), (0.4, 0.5), (-0.2, -0.1), (0.6, 0.7), (-0.4, -0.3)$ — fitting these alone gives a slope of roughly $1.0$ to $1.1$. Now add a sixth point: a microcap that had a huge, signal-unrelated earnings-driven jump, with signal $s=0.1$ (near zero, unremarkable) but return $r=8.0$ (a massive outlier). Refitting with this point included pulls the slope estimate sharply — a near-zero-signal, huge-return point acts like a lever far from the rest of the data on the return axis, and depending on its exact leverage can drag the fitted slope down substantially or inflate the residual variance enough to blow out the standard error either way. One observation, out of six, has materially changed both the point estimate and its apparent significance — motivating winsorizing extreme returns or fitting with a robust regression before trusting the result.

What this means in practice

• This is the core discipline behind every credible "this signal has alpha" claim in quant equity research — Fama-MacBeth or clustered-by-date standard errors, winsorizing extreme names, and checking IC stability across sub-periods are the standard responses to exactly these two failure modes.
• Report the effective sample size, not the raw row count. A backtest description that reports "500 stocks x 250 days = 125,000 observations" as if that were the statistical sample size is a red flag worth checking, not a badge of rigor.
• Watch microcaps and post-earnings jumps specifically — they are disproportionately likely to be the single influential points that swing a cross-sectional slope, and value-weighted or robust-regression variants of the same test often tell a materially different, more honest story.