The EM Algorithm

Some of the most useful models in finance are built on something you never observe. A regime model says the market is either calm or turbulent — but nobody publishes which one today was. A mixture model says returns come from two populations — but each return arrives unlabelled. A dataset has holes where a price failed to print. In every case the parameters are easy to compute if only you knew the hidden piece, and the hidden piece is easy to guess if only you knew the parameters. That circle is what the EM algorithm breaks.

An analogy: the school photo with no names

A school measures the heights of a hundred children from two year groups, then loses the record of which group each child was in. You are asked for the average height of each group.

Hand me the missing labels and the job takes ten seconds: sort the children, average each pile. Hand me the two group averages instead and guessing the labels is nearly as easy: a very tall child is probably older, a very short one probably younger, a mid-height child genuinely uncertain — maybe 60% older, 40% younger.

EM exploits the fact that either half is easy. Guess the two averages. Use that guess to give every child a soft label — not "older" but "70% older" — the E-step (expectation). Then recompute each group's average, letting every child contribute in proportion to their soft label — the M-step (maximisation). The new averages beat the ones you started with. Repeat.

The crucial word is soft. A child who could belong to either group is not forced into one; they contribute partially to both. That is what stops the algorithm thrashing between assignments, and it is the difference between EM and its harder-edged cousin, k-means.

The two steps, one symbol at a time

Write $x_i$ for the $i$-th observation, $k$ for a component (calm or turbulent, younger or older), $\pi_k$ ("pi sub k") for the share of the data you currently believe belongs to component $k$, and $f_k(x)$ for the density of component $k$ — how likely that component is to produce a value near $x$.

E-step. For each observation, compute its responsibility, written $r_{ik}$: the probability that observation $i$ came from component $k$, given everything you currently believe.

$$r_{ik} = \frac{\pi_k f_k(x_i)}{\sum_j \pi_j f_j(x_i)} .$$

In plain English: how likely is this component to have produced this point, relative to all components combined? This is nothing but Bayes' theorem — prior share times likelihood, normalised so the responsibilities across components add to one.

M-step. Refit each component using every data point, weighted by its responsibility:

$$\pi_k^{\text{new}} = \frac{1}{n}\sum_{i=1}^{n} r_{ik}, \qquad \mu_k^{\text{new}} = \frac{\sum_i r_{ik}\, x_i}{\sum_i r_{ik}} .$$

In plain English: the new share of a component is the average responsibility it claimed, and its new centre is a weighted average of the data, where points that probably belong to it count fully and points that probably do not count barely at all.

Then loop. The guarantee that makes EM more than a hopeful heuristic is this: every iteration leaves the likelihood at least as high as it was. The algorithm can never make the fit worse, so it always converges to something.

Worked example 1: filling in missing data

The cleanest possible case. You have five daily P&L figures but two were never recorded. The three you have are 4, 6 and 8, and you want the mean of all five.

Start from a deliberately bad guess, $\mu_0 = 0$.

E-step. Your best estimate of a missing value is the current mean, so fill both holes with 0. M-step. Recompute the mean of the completed data: $(4 + 6 + 8 + 0 + 0)/5 = 3.6$.

Round two:

E-step. Fill both holes with 3.6. M-step. $(18 + 3.6 + 3.6)/5 = 25.2/5 = 5.04$.

Keep going. Each round the update is $\mu_{\text{new}} = (18 + 2\mu_{\text{old}})/5$:

$$0 \to 3.60 \to 5.04 \to 5.62 \to 5.85 \to 5.94 \to 5.98 \to \cdots \to 6$$

It converges to 6, the average of the three observations you actually had — the right answer, since the missing days carry no information about the mean. Notice the shape of the climb: big steps first, then progressively smaller. That slow crawl at the end is EM's signature, and the reason people complain about its speed.

Worked example 2: one full round on a two-regime model

Now the version quants actually use. Model daily returns as a mixture of a calm component (mean 0, standard deviation 1%) and a turbulent one (mean 0, standard deviation 3%), currently believed to be 80% and 20% of days.

Five days of returns, in percent: $0.2,\ -0.5,\ 0.3,\ -3.0,\ 0.8$.

E-step, one point in full. For the $-3.0$ day, the normal density at $-3$ is $0.00443$ under the calm component and $0.0807$ under the turbulent one. Weight each by its share:

$$0.8 \times 0.00443 = 0.00355, \qquad 0.2 \times 0.0807 = 0.01613 .$$

The responsibility of the turbulent component is its share of the total:

$$r_{\text{turb}} = \frac{0.01613}{0.00355 + 0.01613} = \frac{0.01613}{0.01968} = 0.82 .$$

A $-3\%$ day is 82% turbulent, even though turbulent days are only a fifth of the sample. Rarity in the tails beats the prior.

E-step, the rest. The same arithmetic on the other four days gives:

day	return	$r_{\text{turb}}$
1	$+0.2$	0.078
2	$-0.5$	0.085
3	$+0.3$	0.080
4	$-3.0$	0.820
5	$+0.8$	0.100

M-step. The new turbulent share is the average responsibility:

$$\pi_{\text{turb}}^{\text{new}} = \frac{0.078 + 0.085 + 0.080 + 0.820 + 0.100}{5} = \frac{1.163}{5} = 0.233 .$$

Up from 0.20. And the new calm variance is a responsibility-weighted average of the squared returns, using the calm weights $1 - r_{\text{turb}}$:

$$\sigma_{\text{calm}}^2 = \frac{2.546}{3.837} = 0.664 \quad \Longrightarrow \quad \sigma_{\text{calm}} = 0.81\% .$$

The calm component tightened, because the wild day was mostly handed to the turbulent component and stopped inflating the quiet one. That is the value of the method in one line: it separates two populations that arrived mixed together, using nothing but the data.

Five days is far too few for the estimates to settle; with hundreds the updates stop lurching. Drag the width slider below to feel the mechanism — a wide bell puts far more density in the tails than a narrow one, and that density ratio is what the E-step weighs.

What this means in practice

• Regime models run on EM. The Baum-Welch algorithm that fits hidden Markov models is EM with the hidden states as the missing data. Every two-state calm/crisis volatility model you have seen was fitted this way.
• State-space models too. Pairing a Kalman smoother (the E-step) with a closed-form parameter update (the M-step) is the standard way to estimate an unobserved-factor model.
• Missing data without deletion. EM lets you use incomplete rows instead of dropping them, which matters when the gaps are not random — a missing price during a crisis is exactly the observation you cannot afford to discard.
• Always restart it. Run the fit from several different random starting points and keep the best. This is not optional hygiene; it is the standard defence against the failure below.