Particle Filters and Sequential Monte Carlo

Suppose you're tracking a hidden regime — say, whether a market is currently in a "trending" or "mean-reverting" state — and you only get noisy hints about it from observed price behavior, never a direct readout. If the relationship between the hidden state and what you observe is linear and everything is normally distributed, the Kalman filter gives you an exact, elegant answer. But real markets are messier: regime switches happen abruptly (not smoothly), volatility has fat tails, and the true state might follow rules a Kalman filter's linear-Gaussian machinery can't represent. Particle filters solve this more general problem by brute force simulation: instead of tracking one clean estimate, track a swarm of thousands of guesses ("particles"), each representing a plausible version of the hidden state, and let the data itself reshape the swarm over time.

An analogy: a search party with flashlights

Imagine searching for someone lost in the woods at night with no direct sightline, only faint sounds. Instead of committing to one guess of their location, you send out hundreds of searchers (particles) spread across plausible locations, each carrying a flashlight. Each time a new clue arrives (a distant shout, a rustle), searchers whose current position is more consistent with that clue "survive" and get more weight, while those in unlikely spots fade out and get replaced by clones of the more plausible searchers, resampled and nudged forward in time by a bit of random wandering (to represent how the lost person might have moved). Over many rounds of clues, the swarm of searchers naturally concentrates around the true, most plausible location — without ever needing a single clean mathematical formula for "where they are."

The mechanics, one symbol at a time

A particle filter maintains $N$ particles $\{x_t^{(i)}\}_{i=1}^{N}$, each a candidate value for the hidden state at time $t$, each carrying a weight $w_t^{(i)}$ reflecting how consistent it is with the data observed so far. At each new observation $y_t$, three steps repeat:

$$x_t^{(i)} \sim p(x_t \mid x_{t-1}^{(i)}) \quad \text{(propagate)}, \qquad w_t^{(i)} \propto w_{t-1}^{(i)}\, p(y_t \mid x_t^{(i)}) \quad \text{(reweight)}.$$

In plain English: first, propagate each particle forward one step according to whatever model governs how the hidden state evolves (adding some random wander, since the future state is uncertain); second, reweight each particle by how likely the actual new observation $y_t$ would have been if that particle's state were true — particles that "predicted well" get heavier weight, particles that predicted poorly get lighter. A third step, resampling, periodically discards low-weight particles and duplicates high-weight ones, so computational effort concentrates where the probability mass actually is rather than being wasted tracking implausible states. The final estimate of the hidden state at any time is a weighted average (or full weighted distribution) across all surviving particles.

Worked example 1: tracking hidden volatility with 5 particles (illustrative)

Suppose 5 particles currently represent candidate values of hidden volatility: $\{0.8\%, 1.0\%, 1.2\%, 1.5\%, 2.0\%\}$, each starting with equal weight $0.20$. A new, unusually large return arrives, which is much more consistent with a high-volatility state. Suppose the likelihood of observing this return under each particle's volatility works out to relative weights $\{0.02, 0.05, 0.13, 0.30, 0.50\}$ (unnormalized). Normalizing (dividing by the sum, $1.00$): $\{0.02, 0.05, 0.13, 0.30, 0.50\}$ already sums to 1.00 here, so these are the new weights. The weighted average state estimate:

$$\hat{x}_t = 0.02(0.8) + 0.05(1.0) + 0.13(1.2) + 0.30(1.5) + 0.50(2.0) = 0.016+0.05+0.156+0.45+1.0 = 1.67\%.$$

The estimate has shifted sharply toward the higher-volatility particles, exactly reflecting that the observed large return was much more consistent with a high-volatility hidden state than a low one.

Worked example 2: why resampling matters

Suppose after several more rounds without resampling, weights degenerate to $\{0.001, 0.001, 0.002, 0.006, 0.990\}$ — nearly all the probability mass sits on a single particle, and the other four are contributing almost nothing to the estimate despite still consuming computation. This is "particle degeneracy." Resampling addresses it directly: draw 5 new particles from the current weighted set, with each particle's chance of being drawn proportional to its weight — in this case, the particle at 2.0% would be selected roughly 4 to 5 times out of 5 draws, replacing the four nearly-dead particles with copies of the dominant one (each then given a small independent random nudge going forward), refreshing the swarm's diversity around the currently most plausible region instead of wasting effort tracking dead ends.

What this means in practice

Particle filters are used wherever a hidden state must be tracked through a nonlinear or non-Gaussian relationship that a Kalman filter can't handle exactly — regime-switching models, stochastic volatility models with fat-tailed shocks, and option-implied state estimation. They trade the Kalman filter's closed-form exactness for flexibility, at the cost of computational expense (many particles, many resampling steps) and some Monte Carlo noise in the estimates.