The Law of Total Variance

Total portfolio variance seems like it should come purely from day-to-day price wiggles. But regimes matter too — a strategy might be calm within any given market regime (say, low-vol vs high-vol) yet still show large overall variance simply because it behaves very differently across regimes. The law of total variance is the exact identity that splits total variance into these two separate sources, and it is the reason "average within-group variance" is never the whole risk story.

The analogy: variance within classrooms versus variance between classrooms

Imagine measuring the height of every student in a school, split across ten classrooms. Some spread comes from ordinary kid-to-kid variation within any one classroom. But some spread also comes from the fact that a fifth-grade classroom's average height is simply different from a first-grade classroom's average height — spread between classroom averages. Add a student's classroom-level average height back to the picture, and total height variance in the whole school splits cleanly into "how much kids vary around their own classroom's average" plus "how much classroom averages vary from each other." Neither piece alone tells the whole story; you need both.

Writing it down

Let $X$ be the quantity you care about (say, a strategy's daily return) and $Y$ the "grouping" variable (say, which volatility regime today falls into). The law of total variance says

$$\operatorname{Var}(X) = E\big[\operatorname{Var}(X \mid Y)\big] + \operatorname{Var}\big(E[X \mid Y]\big) .$$

Read the two terms in words. The first term, $E[\operatorname{Var}(X\mid Y)]$, is the average within-group variance: for each fixed regime $Y=y$, compute how much $X$ wiggles inside that regime, then average that wiggle across regimes, weighted by how likely each regime is. The second term, $\operatorname{Var}(E[X\mid Y])$, is the variance of the group means: compute each regime's average return, then measure how much those regime-averages themselves spread out. Total variance is the sum of "typical spread inside a group" and "spread of the group centers" — nothing is double-counted, and nothing is left out.

Worked example 1: two regimes, computed by hand

Suppose daily returns come from two regimes, each equally likely ($P(Y=\text{calm}) = P(Y=\text{stormy}) = 0.5$). In "calm," returns average $0\%$ with variance $1$ (in %² units). In "stormy," returns average $-2\%$ with variance $9$. Compute the two pieces.

Within-group term: $E[\operatorname{Var}(X\mid Y)] = 0.5 \times 1 + 0.5 \times 9 = 0.5 + 4.5 = 5.0$.

Between-group term: first find the overall mean, $E[X] = 0.5\times 0 + 0.5 \times(-2) = -1$. The group means are $0$ and $-2$; their variance around the overall mean $-1$ is $0.5 \times (0-(-1))^2 + 0.5 \times (-2-(-1))^2 = 0.5(1) + 0.5(1) = 1.0$.

Total: $5.0 + 1.0 = 6.0$. Check it directly by computing $\operatorname{Var}(X)$ from the full mixture — a two-point mixture of normals with those means and variances gives exactly $\operatorname{Var}(X) = 6.0$ by the same decomposition, confirming the identity holds exactly, not approximately.

Worked example 2: why ignoring the between term understates risk

A risk manager estimates "typical" daily variance by averaging each regime's variance, getting $5.0$ from example 1, and reports that as the portfolio's variance. The true total variance is $6.0$ — a 20% underestimate. The missing $1.0$ is entirely due to regime-switching risk: even if each regime were perfectly calm internally, the fact that the mean return jumps between $0\%$ and $-2\%$ as regimes change is itself a source of variance that a within-group-only estimate silently drops. Doubling the mean gap — say calm at $0\%$ and stormy at $-4\%$ instead of $-2\%$ — would quadruple the between-group term to $4.0$, since it is a squared distance, while leaving the within-group term at $5.0$ unchanged, so total variance would jump to $9.0$.

What this means in practice

Regime-switching risk models, hierarchical Bayesian forecasts, and factor models that pool data across sectors or time periods all rely on this decomposition to separate "noise within a bucket" from "risk from which bucket you're in." A model that only estimates within-group variance — common when volatility is calibrated separately per regime without accounting for how regimes themselves shift — systematically understates total risk, exactly as in worked example 2.