Effective Sample Size With Autocorrelated Observations

Every standard error formula you meet in an introductory statistics course divides by $\sqrt{N}$ and quietly assumes the $N$ observations are independent. Daily P&L, daily prices, anything measured close together in time rarely obeys that assumption — today's number is a partial echo of yesterday's. When that happens, $N$ overstates how much independent evidence you actually have, and every confidence interval built on it is too narrow.

An analogy

Ask 100 strangers on a street their opinion and you get something close to 100 independent data points. Ask the same question to 100 members of one large family and you get something much closer to one opinion, echoed with small variations, because family members influence each other. The headcount is identical; the information is not. Autocorrelated time series are the second case: each new observation is partly an echo of the ones just before it, so the run of $N$ numbers behaves like a smaller number of genuinely fresh observations.

The formula

For a series with correlation only between neighbours (an AR(1) process with correlation $\rho$ between one observation and the next), the effective sample size has a clean closed form:

$$N_{\text{eff}} = N \cdot \frac{1-\rho}{1+\rho}.$$

In words: start with the raw count and multiply by a shrinkage factor that depends only on how strongly each observation echoes the last one. At $\rho = 0$ the factor is 1 and nothing is lost. As $\rho \to 1$ — each observation almost identical to its predecessor — the factor collapses toward zero, because a highly persistent series of any length is carrying only a little genuinely new information. For dependence that stretches across more than one lag, the general version sums over all the autocorrelations $\rho_k$:

$$N_{\text{eff}} = \frac{N}{1 + 2\sum_{k=1}^{N-1}\left(1-\frac{k}{N}\right)\rho_k}.$$

The AR(1) formula above is the special case where only $\rho_1 = \rho$ is nonzero. This is the same quantity that a Newey-West HAC Standard Errors correction is implicitly restoring: instead of shrinking $N$ directly, Newey-West widens the standard error by the equivalent amount.

Worked example

A strategy's daily P&L over one trading year, $N = 252$ days, has lag-one autocorrelation $\rho = 0.30$ — plausible for a book with some overnight carry-over in its positions or marks.

$$N_{\text{eff}} = 252 \times \frac{1-0.30}{1+0.30} = 252 \times \frac{0.70}{1.30} = 252 \times 0.538 = 135.7.$$

A standard error computed as if $N = 252$ would be too small by a factor of $\sqrt{252/135.7} = \sqrt{1.857} = 1.36$ — every $t$-statistic on that P&L series is overstated by roughly 36%. Push the autocorrelation to $\rho = 0.70$, plausible for smoothed or stale marks in an illiquid book, and $N_{\text{eff}} = 252 \times (0.30/1.70) = 44.5$ — barely a fifth of the raw day count, and the naive $t$-statistic is inflated by $\sqrt{252/44.5} = 2.38$-fold.

Why weekly resampling isn't a free fix

A tempting shortcut is to resample the series to a lower frequency — weekly instead of daily — on the theory that widely spaced observations must be less correlated. That helps, but it isn't free: you also discard information every time you resample. Going from 252 daily points to 52 weekly points at $\rho=0.30$ trades a raw count of 252 (worth 135.7 effective observations, computed above) for a raw count of 52 at whatever residual weekly autocorrelation remains, typically well under 52 once the weekly series' own smaller correlation is applied. In this example resampling roughly breaks even at best and can leave you worse off; the honest fix is correcting the standard error at the native frequency, not changing the frequency and hoping the problem disappears.

In practice

• This is why smoothed or stale-marked return series look deceptively good. Illiquid credit, private-market, or lightly-traded book marks that only partially update each day inherit high autocorrelation, which mechanically inflates $N_{\text{eff}}$-blind Sharpe ratios and $t$-statistics.
• Check autocorrelation before trusting any daily-frequency significance test. A quick lag-1 autocorrelation of the return series tells you roughly how much to discount the raw day count.
• The same shrinkage, viewed cross-sectionally rather than across time, is exactly what Estimating Breadth With Correlated Bets computes — correlated bets and correlated time periods both waste headcount in the same mathematical way.
• Overlapping-window regressions are a manufactured, extreme case of this problem; see Overlapping Returns and the Hansen-Hodrick Correction for the version where you built the correlation in yourself by sampling overlapping windows.