Gaussian Mixture Models

Plot a histogram of a stock's daily returns and you rarely get one clean bell. You get a tall narrow spike around zero with long straggling tails, a shape no single normal curve can match: fit the width to the spike and the tails are impossible, fit the width to the tails and the spike is far too short.

The reason is usually that you are not looking at one population. You are looking at two mixed together. Most days the market is calm and returns are tiny; a minority of days it is stressed and returns are large in either direction. Each regime on its own is roughly bell-shaped. Pooled, they make a shape that no bell can copy.

A Gaussian mixture model takes that story literally. It says the data comes from $K$ separate normal distributions, and each observation was drawn from one of them, you just do not get told which.

The model

$$p(x) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x \mid \mu_k, \sigma_k^2)$$

In words: the chance of seeing a value $x$ is a weighted average of $K$ bell curves. Component $k$ has centre $\mu_k$, width $\sigma_k$, and a mixing weight $\pi_k$ which is the fraction of the population it accounts for. The weights are all positive and sum to one, so they behave exactly like probabilities of picking that component.

Notice what the mixture bought you: a distribution that is simultaneously more peaked and more fat-tailed than a normal, built entirely out of normals. That is why mixtures are the standard cheap way to model returns without abandoning Gaussian machinery.

Responsibilities: the soft label

Since the label is hidden, the model reports a probability instead. For point $x$, the responsibility of component $k$ is

$$r_k(x) = \frac{\pi_k \, \mathcal{N}(x \mid \mu_k, \sigma_k^2)}{\sum_{j} \pi_j \, \mathcal{N}(x \mid \mu_j, \sigma_j^2)}$$

In words: how much of the total probability of seeing this point came from component $k$. It is Bayes' rule with the mixing weights as the prior. The responsibilities for one point always add to one.

Worked example: which regime was that day?

Fit a two-component mixture to daily returns in percent and suppose it lands on:

• calm: $\pi_1 = 0.8$, $\mu_1 = 0.05$, $\sigma_1 = 0.6$
• stressed: $\pi_2 = 0.2$, $\mu_2 = -0.3$, $\sigma_2 = 2.4$

Today the market fell $2\%$, so $x = -2.0$. Evaluate each bell at that point using $\mathcal{N} = \frac{1}{\sigma\sqrt{2\pi}} e^{-(x-\mu)^2 / 2\sigma^2}$.

Calm component. The gap is $-2.0 - 0.05 = -2.05$, which is $-2.05 / 0.6 = -3.42$ standard deviations. So the exponent is $-3.42^2/2 = -5.84$ and $e^{-5.84} = 0.0029$. The leading constant is $1/(0.6 \times 2.5066) = 0.665$, giving a density of $0.665 \times 0.0029 = 0.00194$. Weighted: $0.8 \times 0.00194 = 0.00155$.

Stressed component. The gap is $-1.7$, which is only $-0.71$ standard deviations. The exponent is $-0.25$, $e^{-0.25} = 0.778$, and the constant is $1/(2.4 \times 2.5066) = 0.166$, giving $0.129$. Weighted: $0.2 \times 0.129 = 0.02587$.

Add them for the mixture density, $0.00155 + 0.02587 = 0.02742$, then divide:

$$r_2 = \frac{0.02587}{0.02742} = 0.94 .$$

A $-2\%$ day is 94% stressed, 6% calm. Run the same arithmetic on a quiet $+0.3\%$ day and the numbers flip: the calm component contributes $0.488$, the stressed one $0.032$, and the day comes out 94% calm. The model never says "regime 2" outright; it hands you a number you can act on proportionally.

Fitting, and choosing K

You cannot maximise the likelihood in closed form, because the sum inside the logarithm does not simplify. The standard fix is the The Expectation-Maximization Algorithm: guess parameters, compute responsibilities, then re-estimate each component's mean and variance using those responsibilities as weights, and repeat. Each pass provably does not decrease the likelihood.

$K$ is yours to choose. Adding components always improves the fit on the data you fitted, so use a penalised criterion such as BIC, or out-of-sample likelihood, rather than eyeballing the histogram.

Where quants use it

Two-state mixtures are the workhorse of Regime Detection — a calm component and a crisis component, with the responsibility acting as a continuously updated "probability we are in a stress regime" that can scale position size. Mixtures also give a tractable fat-tailed return distribution for Value at Risk (VaR) work, and in multiple dimensions each component carries its own covariance matrix, so a GMM can find clusters that are elongated and tilted rather than round.