The Bellman Equations

The value of a state is the total discounted reward you expect from it onward — an infinite sum over every possible future. Written out directly it is hopeless: to know what today is worth you need to know what tomorrow is worth, which needs the day after, forever. Richard Bellman's contribution was to notice that this circularity is not a bug. Turn it into an equation and it becomes the most useful identity in sequential decision-making.

The analogy: how long to the airport

Someone asks how long it takes to reach the airport from your front door. You do not simulate every possible route to the terminal. You say: about four minutes to the motorway junction, and from that junction it is about forty. Total, forty-four.

That answer has the Bellman shape. The time from here equals the time to take one step, plus the time from wherever that step lands you. You have replaced an enormous question ("the whole journey") with a small one ("one hop") plus a reference to the same kind of answer at the next place. Two more properties make it work: if the best route to the airport passes through that junction, then the rest of it must be the best route from the junction — no shortcut you would refuse to take later helps you now. And because each remaining leg is shorter than the last, repeatedly applying the rule pins the numbers down.

The Bellman expectation equation

Fix a policy $\pi$. The value $v_\pi(s)$ of state $s$ satisfies

$$v_\pi(s) = \sum_{a} \pi(a \mid s) \sum_{s'} P(s' \mid s, a) \Big[ R(s, a) + \gamma \, v_\pi(s') \Big].$$

Read it inside out. The bracket is "the reward you collect for this step, plus $\gamma$ times whatever the next state is worth". The inner sum averages that over the states you might land in, weighted by the transition probabilities $P$. The outer sum averages over the actions your policy might take. Nothing more complicated than a two-level weighted average.

Worked example: solving a two-state market

A crude regime model. The market is either quiet or turbulent. Holding a position pays $+1$ per step when quiet and $-2$ per step when turbulent. From quiet, you stay quiet with probability $0.8$ and turn turbulent with $0.2$. From turbulent, it is an even split back to quiet or staying turbulent. Take $\gamma = 0.9$.

Write the Bellman equation once per state:

$$v(Q) = 1 + 0.9\big[0.8\, v(Q) + 0.2\, v(T)\big], \qquad v(T) = -2 + 0.9\big[0.5\, v(Q) + 0.5\, v(T)\big].$$

Multiply out and collect terms:

$$0.28\, v(Q) - 0.18\, v(T) = 1, \qquad -0.45\, v(Q) + 0.55\, v(T) = -2 .$$

Two linear equations, two unknowns. The determinant is $0.28(0.55) - (0.18)(0.45) = 0.154 - 0.081 = 0.073$, so

$$v(Q) = \frac{0.55 - 0.36}{0.073} = \frac{0.19}{0.073} = 2.60, \qquad v(T) = \frac{-0.56 + 0.45}{0.073} = \frac{-0.11}{0.073} = -1.51 .$$

Check the first one: $1 + 0.9[0.8(2.60) + 0.2(-1.51)] = 1 + 0.9(1.78) = 2.60$. It holds.

Two lessons hide in those numbers. Being quiet is worth $2.60$, not the $1/(1-0.9) = 10$ you would get if quiet lasted forever — the $20$ percent chance of turning turbulent each step costs you three quarters of the value. And being turbulent is worth $-1.51$, much better than the $-20$ of permanent turbulence, because you expect to escape quickly. The value function prices the whole future, not the present state.

The optimality equation

If you get to choose the action rather than follow a fixed policy, the outer average becomes a maximum:

$$v_*(s) = \max_{a} \sum_{s'} P(s' \mid s, a) \Big[ R(s, a) + \gamma \, v_*(s') \Big].$$

In words: the best you can do here is, over all actions available, the one whose average of "reward now plus discounted value next" is largest. The equivalent statement in terms of action values is

$$q_*(s, a) = \sum_{s'} P(s' \mid s, a) \Big[ R(s, a) + \gamma \max_{a'} q_*(s', a') \Big],$$

which is more convenient in practice because once you have $q_*$ the optimal policy is simply "pick the action with the biggest $q$" — no model of $P$ required. That is the observation Q-learning is built on.

The $\max$ makes the system nonlinear, so you cannot solve it with the linear algebra used above. You iterate instead.

Worked example: value iteration, sweep by sweep

Same two-state market. Start from complete ignorance, $v(Q) = v(T) = 0$, and repeatedly apply the equation using the previous sweep's numbers.

Sweep 1. $v(Q) = 1 + 0.9[0.8(0) + 0.2(0)] = 1.00$ and $v(T) = -2 + 0 = -2.00$.

Sweep 2. $v(Q) = 1 + 0.9[0.8(1.00) + 0.2(-2.00)] = 1 + 0.9(0.40) = 1.36$, and $v(T) = -2 + 0.9[0.5(1.00) + 0.5(-2.00)] = -2 - 0.45 = -2.45$.

Sweep 3. $v(Q) = 1 + 0.9[0.8(1.36) + 0.2(-2.45)] = 1 + 0.9(0.598) = 1.54$, and $v(T) = -2 + 0.9(-0.545) = -2.49$.

Sweep 4. $v(Q) = 1 + 0.9[0.8(1.54) + 0.2(-2.49)] = 1.66$, and $v(T) = -2 + 0.9(-0.476) = -2.43$.

The numbers are crawling toward $2.60$ and $-1.51$. Watch the largest error: $1.60$, then $1.24$, $1.07$, $0.94$. Each sweep shrinks it by a factor tending to $\gamma = 0.9$ — that is the contraction property, and it is why value iteration always converges. After 60 sweeps the error is below $2.6 \times 0.9^{60} \approx 0.005$.

What this means in practice

Almost every reinforcement learning algorithm is the Bellman equation with a different way of handling the parts you do not know:

• You know $P$ and $R$: apply the equation directly, as above. That is value and policy iteration, and it is exact.
• You do not know $P$: replace the expectation with a sample. One observed transition gives $R + \gamma v(s')$, and nudging your estimate toward it is temporal difference learning.
• Too many states to tabulate: replace $v$ with a function approximator. The equation is unchanged; the guarantees weaken sharply, because a network fitting one state's value disturbs every other state's.

For a quant, the practical payoff is a way to price a strategy rather than a trade. "What is this position worth?" becomes "what is the value of the state I am in, given how I intend to behave from here?" — which is the right question when your actions today constrain your options tomorrow.