The Expectation-Maximization Algorithm

Fitting a model is easy when you can see everything. If someone hands you a list of daily returns already tagged calm or stressed, estimating the two regimes takes one pass: average the calm days, average the stressed days, done. And if someone hands you the two regimes' parameters, tagging each day is equally easy: work out which regime explains that day better.

The trouble is you get neither. You have the returns and nothing else. You need the labels to fit the parameters, and you need the parameters to work out the labels. That is the deadlock Expectation-Maximization breaks, and it breaks it in the most obvious way imaginable: guess one, derive the other, and go round again.

The two steps

Start from any parameter guess, however crude. Then repeat:

• E-step (Expectation). Hold the parameters fixed and compute, for every data point, the probability that it came from each hidden group. These are soft labels — a point can be 70% one group and 30% the other. They are called responsibilities.
• M-step (Maximization). Hold the responsibilities fixed and refit the parameters as if the soft labels were real, using each point fractionally in each group according to its responsibility.

Stop when the parameters stop moving.

Worked example: two coins, unknown which is which

Two biased coins, A and B, land heads with unknown probabilities $\theta_A$ and $\theta_B$. Someone runs five experiments. Each time they pick a coin at random, flip it ten times, and record only the number of heads, never which coin they used:

experiment	heads	tails
1	5	5
2	9	1
3	8	2
4	4	6
5	7	3

Guess $\theta_A = 0.60$ and $\theta_B = 0.50$ to get started.

E-step. For each experiment, compute how likely that head count is under each coin, then normalise. Take experiment 2, nine heads and one tail:

$$P(\text{data} \mid A) = 0.6^9 \times 0.4^1 = 0.00403, \qquad P(\text{data} \mid B) = 0.5^{10} = 0.00098 .$$

Coin A explains nine heads about four times better than a fair coin does. Normalising, the responsibility of coin A is

$$r_A = \frac{0.00403}{0.00403 + 0.00098} = 0.80 ,$$

so experiment 2 counts as 80% coin A and 20% coin B. Repeating for all five gives A-responsibilities of $0.45,\ 0.80,\ 0.73,\ 0.35,\ 0.65$ — heads-heavy experiments lean towards A, tails-heavy ones towards B, exactly as intuition says.

M-step. Now re-estimate each coin's bias using fractional flips. Coin A's share of the heads is

$$0.45(5) + 0.80(9) + 0.73(8) + 0.35(4) + 0.65(7) = 21.24 ,$$

and its share of the tails, computed the same way, is $8.56$. So

$$\theta_A = \frac{21.24}{21.24 + 8.56} = 0.71 .$$

In plain terms: coin A was credited with 21.24 heads out of 29.8 flips it was probably responsible for. The same arithmetic on coin B gives $11.76$ heads out of $20.2$, so $\theta_B = 0.58$.

One round moved the estimates from $(0.60,\ 0.50)$ to $(0.71,\ 0.58)$. Run it again and again and they settle near $(0.80,\ 0.52)$ — close to what you would have got if the coin labels had never been hidden.

Why it works

Each E-step builds a lower bound on the log-likelihood that touches it exactly at the current parameters. The M-step then jumps to the top of that bound. Since you started on the true curve and moved up the bound, you cannot have gone down on the true curve either. That is the whole proof sketch, and it is why the likelihood trace in the figure only ever climbs.

Where quants meet it

EM is the standard fitting routine for Gaussian Mixture Models, where the hidden label is "which regime produced this return". It is also the engine behind the Baum-Welch algorithm for Hidden Markov Models, the smoothing step in some The Kalman Filter estimation, and a clean way to handle genuinely missing observations — a stock that stopped trading for a week is just another latent quantity to average over rather than a row to delete.