Regression to the Mean

Every year, financial media runs a story about last year's best-performing fund managers — and every year, most of them underperform the following year. The instinctive explanation is that success breeds complacency, or that a hot strategy gets crowded and stops working. Often the real explanation is much simpler and purely statistical: if a manager's yearly return is partly skill and partly luck, the managers who happened to rank at the very top last year were disproportionately the ones who also got lucky, and luck — being random — doesn't repeat. Their next year's return reverts toward their true average skill, which is usually less extreme than the number that got them noticed.

An analogy: exam scores and lucky guesses

Imagine a class where every student's exam score is 80% true knowledge and 20% luck (which questions happened to be ones they'd just reviewed, guessing right on an unfamiliar one). The student who scored highest this time is disproportionately likely to be someone who knew the material and got lucky, since getting both is what it takes to land at the very top. Next exam, that same student's luck resets — it doesn't carry over — so their expected score falls back toward their own true ability, even if their studying habits haven't changed one bit. The student didn't get worse; the extreme luck that pushed them to the top simply wasn't there to repeat.

The mechanics, one piece at a time

Suppose an observed measurement $X$ is the sum of a stable true value $\mu$ (skill, ability, ability to generate alpha) and independent noise $\varepsilon$: $X = \mu + \varepsilon$, with $\varepsilon$ having mean zero. If you select on an extreme value of $X$ (say, the top 5%), the average true $\mu$ among those selected is less extreme than the average observed $X$, because some of what made $X$ extreme was noise, which won't repeat. Formally, if you regress next period's value $Y$ on this period's value $X$, the expected next-period value given an extreme $X$ is:

$$\mathbb{E}[Y \mid X] = \bar{Y} + \rho \cdot (X - \bar{X}),$$

where $\rho$ is the correlation between $X$ and $Y$ across the population and $\bar{X}, \bar{Y}$ are the population averages. In plain English: the best prediction for next period isn't "repeat this period's extreme value" — it's "start at the average, and move only partway toward this period's value, by a fraction $\rho$ that reflects how much of the measurement is signal versus noise." When $\rho$ is small (most of the variation is noise), the prediction reverts almost all the way back to the average; when $\rho$ is close to 1 (the measurement is mostly signal), it stays close to the observed extreme.

Worked example 1: fund manager returns

Suppose a fund manager's annual return is $\mu = 8\%$ true skill plus noise with standard deviation 15%, and the correlation between one year's return and the next is estimated at $\rho = 0.2$ (mostly noise, some skill). Manager A had a 35% return last year — 27 points above the population average of 8%. The best forecast for next year isn't 35%; it's $8\% + 0.2 \times (35\%-8\%) = 8\% + 5.4\% = 13.4\%$ — reverted most of the way back toward the 8% average, because most of that 35% was likely a lucky year, not a permanent 27-point skill edge.

Worked example 2: quantifying the pull with real numbers

A pension fund ranks 200 managers by trailing 3-year return and hires only the top 10. The top 10 averaged 22% annualized, versus a full-universe average of 9%. If the true cross-sectional correlation between one 3-year window and the next is $\rho = 0.3$, the expected next 3-year return for these same top-10 managers is $9\% + 0.3\times(22\%-9\%) = 9\% + 3.9\% = 12.9\%$ — still above average (some real skill signal remains, since $\rho>0$), but far below the 22% that got them hired. A pension fund that budgets for 22% going forward, attributing all of it to skill, has fallen for regression to the mean.

Drag $\rho$ down toward 0 and watch how the top-right points (last period's biggest winners) fall back toward the average on the next-period axis — that pullback is regression to the mean, and it gets stronger as correlation weakens.

What this means in practice

Regression to the mean shows up everywhere performance is measured and selected on: manager selection, backtested strategy rankings, "hot hand" narratives, year-over-year KPI reviews, and even A/B test winners. The practical defense is to never forecast forward using the exact extreme number that got something selected — always shrink it toward the relevant population average by an amount that reflects how noisy the underlying measurement is, exactly as a mixed-effects model does automatically (see Mixed-Effects Models).