Two Worlds, One Series: Hamilton's Regime-Switching Model

Anyone who has traded through a crisis knows a fact that most models refuse to acknowledge: markets have moods. There are calm periods where volatility is low, correlations are modest, and mean reversion works. Then there are panics, where volatility triples, everything correlates to one, and the strategies that worked last month get run over.

The standard time-series model of 1988 could not express this. It assumed one set of parameters, fixed forever. It would fit a single average volatility, a single average drift, and describe neither the calm nor the panic. Averaging a bull market and a crash gives you a description of a world that has never existed.

James Hamilton's 1989 paper gave econometrics a formal, estimable way to say: there is more than one world, we do not know which one we are in, and the data can tell us.

The problem: the business cycle is a switch, not a wave

Hamilton's motivating question was about recessions. The dominant view treated economic output as a single process: a trend plus some wobble around it. Recessions were just periods where the wobble happened to be negative.

But that is not what recessions look like. A recession is not a big negative wobble. It is a period where the entire character of the economy changes. Growth does not just dip below average, it turns negative and stays negative for several quarters. Then it flips back, and stays positive for years. The economy does not oscillate smoothly. It switches between an expansion mode and a contraction mode, and it persists in whichever mode it is in.

That persistence is the key observation. Recessions cluster. If last quarter was a recession, this quarter probably is too. The world has memory about which state it is in, separate from the memory in the data itself.

The key idea via analogy: a coin that decides which dice you roll

Here is the whole model in one picture.

Imagine there are two bags of dice. Bag A is the good-times bag: roll from it and you get mostly positive returns, modest volatility. Bag B is the bad-times bag: mostly negative, wild swings.

Every period, a hidden switch decides which bag you roll from. But the switch is not a fair coin flip. It is sticky. If you rolled from Bag A last period, there is a high probability, say 95%, that you will roll from Bag A again this period. Same for Bag B. This stickiness is what produces long expansions and long recessions rather than a chaotic flicker.

Now here is the catch that makes the problem hard and the paper brilliant. You never see the switch. Nobody rings a bell to announce "we are now in a recession." All you observe are the numbers that come out. Your job is to work backwards: given the sequence of outcomes you have seen, what is the probability that we were in Bag B last month? And in Bag B right now?

Hamilton's contribution was an algorithm that does exactly this. It walks through the data one period at a time, and at each step updates a running belief: "there is a 73% chance we are currently in the recession state." Each new observation nudges that belief. A string of terrible numbers pushes it up. A recovery pushes it down. The stickiness of the switch means the belief does not whipsaw wildly, it evolves.

Crucially, the same machinery lets you estimate everything at once by maximum likelihood: what the two states look like, how sticky each is, and what the probability of being in each state was at every point in history. You feed in a single series of numbers, and it hands you back a hidden narrative.

The word "Markov" in "Markov switching" is just the technical name for the stickiness rule: the probability of tomorrow's state depends only on today's state, not on the whole history of states.

Why it mattered

• It gave recessions a statistical existence. When Hamilton ran his model on US GDP, the "recession probability" series it produced lined up strikingly well with the recession dates that the National Bureau of Economic Research declares by committee, after the fact, using judgement. A purely mechanical model, given nothing but a growth series, recovered the business cycle. That is a genuinely impressive result.
• It made regime thinking rigorous. Traders always talked about regimes. Hamilton made regimes an object you could estimate, test, and forecast with, rather than a vague narrative.
• It exploded into finance. Regime-switching versions of nearly everything followed: regime-switching volatility models, regime-switching asset allocation, regime-switching term structure models. The idea that correlations and volatilities are state-dependent, and that the state is hidden and persistent, is now central to how risk is thought about.
• It handles the thing that breaks other models. A single-regime model fitted across 2006 to 2010 produces parameters that describe neither 2006 nor 2009. A two-regime model can describe both, and tell you which one you are probably in.

The honest limitations

• You must decide how many regimes exist, and there is no clean way to do it. Two? Three? Five? Standard statistical tests for "is there really more than one regime" break down in this setting for technical reasons, and the honest answer is that the choice is often made by judgement, or by trying several and picking the one that looks nicest. That is a lot of researcher discretion, and researcher discretion is how overfitting gets in.
• The regimes may not mean what you want them to mean. The model finds two clusters of behaviour. It does not label them. It is very easy to fit a two-state model, see that one state has high volatility, and declare it "the crisis regime," when in truth the model has just latched onto something else entirely, or onto nothing at all.
• It is much better at telling you where you have been than where you are. The model's estimate of the current regime, using only data up to today, is far noisier than its retrospective estimate using the full sample. In backtests people routinely look at the smoothed, full-sample regime estimates and are dazzled by how well the model called the crisis. That estimate uses future data. Trading on it is not possible. The real-time estimate is much less impressive, and confusing the two is one of the most seductive lookahead biases in quant finance.
• Regime changes are detected late. Because the switch is sticky and the evidence accumulates gradually, the model typically only becomes confident that a regime has changed after the change is well underway. By the time it says "we are in a crash," the crash has happened.
• It assumes the regimes themselves are stable. The characteristics of "crisis" in 1998, 2008 and 2020 were not the same. A model fitted on old crises may not recognise a new kind.

The one-line takeaway

Hamilton showed how to model a world that secretly flips between hidden, persistent states, and how to infer from the data alone which state you are probably in, which gave economics and finance their formal language for regimes, along with a permanent temptation to admire how well the model called a crisis using data it would not have had at the time.