Conditional Expectation

Conditional expectation is the answer to "given what I now know, what should I expect on average?" Plain Expected Value gives one number for a random quantity before you learn anything. Conditional expectation, written $E[Y \mid X]$, gives your updated average for $Y$ once the value of some related variable $X$ is revealed. It's the mathematical form of learning: new information reshapes your best guess.

The subtle, important part is that $E[Y \mid X]$ is itself a random variable, not a single number. For each possible value $x$ that $X$ might take, there's a conditional average $E[Y \mid X = x]$, computed from the conditional distribution of $Y$ given that $x$. Collect those averages into a rule and you get a function of $X$: feed in whatever $X$ turned out to be, read off the corresponding expected $Y$. Once $X$ is observed the answer becomes a number; before that, it's random because $X$ is.

The tower property

The most-used fact about conditional expectation is the law of total expectation, also called the tower property. It says that if you average all the conditional averages, weighting each by how likely its condition is, you get back the plain unconditional average:

$$E\big[\,E[Y \mid X]\,\big] = E[Y] .$$

In words: split the world by the value of $X$, take the average of $Y$ inside each slice, then average those slice-averages by their probabilities, and you recover the overall mean. It's a formal version of "condition on something to simplify, then un-condition at the end," and it turns many gnarly expectation problems into a two-step calculation.

Worked example: total expectation in two steps

A stock's daily return depends on which regime the market is in. In a calm "up" regime, which occurs with probability $0.6$, the average daily return is $+0.5\%$. In a jittery "down" regime, probability $0.4$, the average return is $-0.3\%$. What's the overall expected daily return?

Condition on the regime, then apply the tower property:

$$E[R] = P(\text{up})\,E[R \mid \text{up}] + P(\text{down})\,E[R \mid \text{down}]$$ $$= (0.6)(0.5\%) + (0.4)(-0.3\%) = 0.30\% - 0.12\% = 0.18\% .$$

The conditional expectations $E[R \mid \text{up}] = 0.5\%$ and $E[R \mid \text{down}] = -0.3\%$ are the two values of the random variable $E[R \mid \text{regime}]$; weighting them by the regime probabilities collapses everything to the single unconditional mean $0.18\%$. That two-line move, condition, then average the conditions, is the workhorse pattern.

The best-predictor property

There's a second reason conditional expectation is everywhere: among all possible functions of $X$, the conditional mean $E[Y \mid X]$ is the one that predicts $Y$ with the smallest average squared error. If you must guess $Y$ from $X$ and you'll be scored on squared mistakes, you cannot beat $E[Y \mid X]$. This is exactly why regression targets the conditional mean, and, when the variables are jointly Gaussian (see the The Multivariate Normal Distribution), that best predictor happens to be a straight line, which is what makes linear regression the correct model there rather than merely a convenient one.

Common pitfalls

• It's a function, not a number. $E[Y \mid X]$ varies with $X$; only $E[Y \mid X = x]$, at a specific $x$, is a number. Mixing the two up is the classic beginner slip.
• Mean-independence is weaker than independence. $E[Y \mid X] = E[Y]$ says $X$ doesn't move $Y$'s mean, but $X$ could still change $Y$'s variance or shape. Full independence is a stronger condition than a flat conditional mean.
• Condition on the right thing. Choosing a helpful variable to condition on is an art; a good choice (like the regime above) makes the inner expectations trivial, a bad one leaves you exactly where you started.

Conditional expectation grows straight out of Conditional Probability (average with respect to the conditional distribution), pairs with the The Law of Total Probability as its expectation-flavoured twin, reads slices out of Joint and Marginal Distributions, and underlies martingales, filtering, and nearly every forecasting model in quant finance.