The Kalman Filter

You often want to know something you can't measure directly. The true price of a thinly-traded asset, a rocket's exact position, a slowly-drifting hedge ratio, none of these come with a clean readout. What you get instead is a stream of noisy measurements. The Kalman filter is the classic recipe for turning that noisy stream into a smooth, best-guess estimate of the hidden truth, updated in real time as each new observation arrives.

Its cleverness is that it never stores the whole history. At every step it carries just two things: its current best estimate and how uncertain it is. When a new measurement lands, it folds that in and moves on. This makes it fast, memory-light, and perfect for live systems, which is why it steered the Apollo spacecraft and now runs quietly inside countless trading and tracking systems.

The predict-update loop

The filter cycles through two moves, over and over:

1. Predict. Use your model of how the state evolves to roll your estimate forward one step. Your best guess drifts to where the dynamics say it should go, and, crucially, your uncertainty grows, because prediction is never perfect.
2. Update. A new measurement arrives. Blend it with the prediction. The blend isn't 50/50, it's weighted by which source you trust more. Fold it in and your uncertainty shrinks, because you just learned something.

The weight in that blend is the Kalman gain $K$. For a simple one-dimensional state it is

$$K = \frac{P}{P + R},$$

where $P$ is the variance of your prediction (how unsure you are before the measurement) and $R$ is the variance of the measurement noise (how unreliable the sensor is). The updated estimate then corrects the prediction by a fraction $K$ of the surprise:

$$\hat x_{\text{new}} = \hat x_{\text{pred}} + K\big(y - \hat x_{\text{pred}}\big).$$

Read the gain: if your prediction is rock-solid ($P$ small) the gain is near $0$ and you barely move for a noisy measurement; if the measurement is pristine ($R$ small) the gain is near $1$ and you snap to it. It's an automatic, self-tuning trust dial.

Worked example: one update step

Your current best estimate of a hidden value is $\hat x_{\text{pred}} = 100$, and after the predict step its uncertainty is $P = 9$ (a standard deviation of $3$). A fresh measurement reads $y = 106$, from a sensor with noise variance $R = 3$.

First the gain:

$$K = \frac{P}{P + R} = \frac{9}{9 + 3} = 0.75.$$

The measurement is more precise than your prediction ($R = 3 < P = 9$), so the gain leans toward it. The surprise is $106 - 100 = 6$, and you correct by three-quarters of it:

$$\hat x_{\text{new}} = 100 + 0.75 \times 6 = 104.5.$$

Finally, your uncertainty drops: the updated variance is $(1 - K)\,P = 0.25 \times 9 = 2.25$, a standard deviation of $1.5$, lower than both the prediction's $3$ and the sensor's $\sqrt 3 \approx 1.73$. That's the quiet magic: combining two imperfect sources yields an estimate sharper than either one, exactly the Bayesian logic of pooling independent evidence, run recursively.

Where it fits in quant work

The filter is the engine behind State-Space Models, and a common use is estimating a time-varying hedge ratio in Pairs Trading: the relationship between two assets drifts, so you treat the hedge ratio as a hidden state and let the Kalman filter track it live instead of refitting a static regression on a rolling window.

Where it misleads

• It assumes linear-Gaussian. The plain filter is optimal only when the dynamics are linear and the noise is Gaussian. Real markets are neither; extensions (extended and unscented Kalman filters, particle filters) handle nonlinearity, at more cost and fragility.
• Garbage tuning, garbage output. The filter's behaviour is set by the noise variances you feed it (the process noise and measurement noise). Set the process noise too low and it grows overconfident and ignores real changes; too high and it chases every wiggle. These knobs must be tuned or estimated, not guessed.
• It trusts its model. If the true dynamics break from your assumed model (a regime shift, a structural break), the filter can lag badly or diverge, its uncertainty says it's confident right when it's most wrong.

The picture to carry: the Kalman filter is a running argument between what you expected and what you just saw, settled every step by whichever is less uncertain. It's Bayesian Inference made recursive, and the workhorse of estimating hidden, drifting quantities from noisy data.