Baum-Welch and Forward-Backward Inference

A hidden Markov model assumes an unobserved state — say, "bull regime" or "bear regime" — evolves over time and produces the observed data (returns, volumes) you actually see, but you don't get to watch the state itself, only its noisy fingerprints. Fitting such a model means simultaneously figuring out the transition probabilities between hidden states and the emission probabilities linking each state to observations, without ever directly observing which state generated which observation — a chicken-and-egg problem, since you'd estimate the parameters easily if you knew the state sequence, and infer the state sequence easily if you knew the parameters.

Baum-Welch is the expectation-maximization algorithm specialized to this setting. The forward-backward algorithm (the "E-step") computes, for every time point, the probability of being in each hidden state given the entire observed sequence — forward passes accumulate evidence from the past, backward passes from the future, and combining them gives the full-sequence state probabilities. The "M-step" then re-estimates transition and emission parameters using those probabilities as soft state-assignment weights, and the two steps alternate, provably increasing the data's likelihood each round, until the parameters stop changing meaningfully.

Fitting a two-state (calm/turbulent) HMM to daily returns starts from a rough guess of transition and emission parameters, runs forward-backward to get each day's probability of being "turbulent," re-estimates the parameters using those probabilities as weights, and repeats — typically converging within a few dozen iterations.