Baum-Welch and Forward-Backward Inference

A hidden Markov model assumes an unobserved state, say, "bull regime" or "bear regime", evolves over time and produces the observed data (returns, volumes) you actually see, but you don't get to watch the state itself, only its noisy fingerprints. Fitting such a model means simultaneously figuring out the transition probabilities between hidden states and the emission probabilities linking each state to observations, without ever directly observing which state generated which observation, a chicken-and-egg problem, since you'd estimate the parameters easily if you knew the state sequence, and infer the state sequence easily if you knew the parameters.

Baum-Welch is the expectation-maximization algorithm specialized to this setting. The forward-backward algorithm (the "E-step") computes, for every time point, the probability of being in each hidden state given the entire observed sequence, forward passes accumulate evidence from the past, backward passes from the future, and combining them gives the full-sequence state probabilities. The "M-step" then re-estimates transition and emission parameters using those probabilities as soft state-assignment weights, and the two steps alternate, provably increasing the data's likelihood each round, until the parameters stop changing meaningfully.

Fitting a two-state (calm/turbulent) HMM to daily returns starts from a rough guess of transition and emission parameters, runs forward-backward to get each day's probability of being "turbulent," re-estimates the parameters using those probabilities as weights, and repeats, typically converging within a few dozen iterations.