Stochastic Control and the HJB Equation

Suppose you manage a portfolio and can rebalance continuously between a risky asset and cash. How much should you hold in the risky asset, at every instant, for the rest of your life, to maximise how happy you'll be with your final wealth? You can't just pick one number — the right allocation depends on how much wealth you currently have, how much time is left, and the fact that tomorrow's price is random. Answering "what should I do right now" when the future keeps branching randomly is what stochastic control is for, and the Hamilton-Jacobi-Bellman (HJB) equation is the single equation that answers it.

The analogy: a canoeist crossing a river with a current that shifts by the minute

Picture paddling across a river to reach a dock, but the current's strength and direction drift randomly as you go. You can't plan your whole route in advance — a plan made now will be wrong five minutes from now, when the current has changed. The only sane strategy is a rule: at every point in the river, given how the current feels right now, choose the paddling direction that gives you the best expected outcome from here onward. That rule is a function of your position and the current, not a fixed list of moves. The HJB equation is the mathematical machine that produces exactly this kind of rule for any random system: instead of planning the whole journey, it tells you the best action at every possible state, at every moment, accounting for all the randomness still to come.

Writing it down

Let $X_t$ be the state of the system at time $t$ — in the canoe story, your position; in finance, your wealth. You pick a control $u_t$ at each instant — your paddling direction, or the fraction of wealth in stocks — and the state evolves as a stochastic differential equation:

$$dX_t = b(X_t, u_t)\,dt + \sigma(X_t, u_t)\,dW_t .$$

In words: the state drifts by an amount $b$ that depends on where you are and what you do, plus a random wiggle of size $\sigma$ driven by Brownian motion $W_t$, whose size also depends on your state and your choice.

You want to choose $u_t$ at every instant to maximise the expected value of a reward — some running payoff $f$ collected along the way plus a payoff $g$ at the final time $T$:

$$V(t,x) = \max_{u} \; \mathbb{E}\left[\int_t^T f(X_s,u_s)\,ds + g(X_T) \;\middle|\; X_t = x\right].$$

In words: $V(t,x)$, the value function, is the best expected total payoff you can achieve if you're in state $x$ at time $t$ and play optimally from there on. It is the entire point of the exercise — everything else builds toward computing $V$.

The HJB equation is what $V$ must satisfy. It comes from a one-line argument: the best thing to do over a tiny interval $dt$ is to pick the $u$ that maximises the instant's reward plus the expected change in the future's best value, and by Itô's lemma that expected change is $\partial_t V + b\,\partial_x V + \tfrac12\sigma^2\partial_{xx}V$. Setting the best achievable rate of improvement to zero (since $V$ is already optimal) gives:

$$\partial_t V(t,x) + \max_{u}\left[f(x,u) + b(x,u)\,\partial_x V(t,x) + \tfrac{1}{2}\sigma(x,u)^2\,\partial_{xx} V(t,x)\right] = 0 .$$

In words: at every instant, the rate the value function is falling due to time passing ($\partial_t V$) is exactly offset by the best you can currently extract — the running reward $f$, plus how a small nudge in the state moves value ($\partial_x V$, the marginal value of the state), plus a correction for the randomness itself ($\partial_{xx}V$, curvature — this is the same Itô correction that shows up everywhere randomness meets calculus). The maximisation inside the brackets is solved pointwise, for each $(t,x)$ separately — that is what turns an infinite-dimensional planning problem into something you can actually compute.

Drag the drift and volatility here and watch how the paths fan out. HJB is the tool that decides, at every point along a fan like this, what action keeps the expected best outcome highest — not just for one path, but averaged over every path that could still happen.

Worked example 1: the merton problem, one step at a time

Take the classic setup: wealth $X_t$, invest fraction $u$ in a risky asset with expected return $\mu$ and volatility $\sigma$, rest in cash earning zero, no consumption, maximise $\mathbb{E}[\log X_T]$. Wealth evolves as $dX = u\mu X\,dt + u\sigma X\,dW$. Guess $V(t,x) = \log x + h(t)$ for some function $h$ (log utility has this clean separable form). Then $\partial_x V = 1/x$, $\partial_{xx}V = -1/x^2$, and the HJB equation becomes

$$h'(t) + \max_u \left[u\mu - \tfrac12 u^2\sigma^2\right] = 0 .$$

The bracket is a plain quadratic in $u$ — maximise it by calculus: differentiate with respect to $u$, set to zero: $\mu - u\sigma^2 = 0$, so $u^\star = \mu/\sigma^2$. Plug numbers in: $\mu = 8\%$, $\sigma = 20\%$, so $u^\star = 0.08 / 0.04 = 2.0$. This says invest 200% of wealth in the risky asset — borrowing 100% of wealth to lever up — because log utility with this risk-reward ratio wants that much exposure. Note the answer $u^\star$ didn't depend on wealth $x$ or time $t$ at all; that constant-fraction result is the celebrated Merton finding, and it dropped straight out of solving the bracket.

Worked example 2: a two-state grid, done by hand

Take a toy two-period, two-state version to see the "solve backward" logic without calculus. Wealth is either $90 or $110 at time 1 (equally likely), and at time 1 you choose to hold cash (locks in current wealth) or a coin-flip bet that pays $20 gain or $20 loss with equal probability. Utility is $\sqrt{\text{wealth}}$.

At $90: cash gives $\sqrt{90}=9.487$. The bet gives $0.5\sqrt{110}+0.5\sqrt{70}=0.5(10.488)+0.5(8.367)=9.427$. Cash wins, so $V(90)=9.487$.

At $110: cash gives $\sqrt{110}=10.488$. The bet gives $0.5\sqrt{130}+0.5\sqrt{90}=0.5(11.402)+0.5(9.487)=10.444$. Cash wins again, $V(110)=10.488$.

Working backward from these values to time 0 is exactly the HJB logic in discrete form: at each state, compare the payoff of each action plus the already-computed future value, and keep the best. The continuous-time HJB equation is the same backward sweep, shrunk to instants and solved with calculus instead of a lookup table.

What this means in practice

• Portfolio choice, market making, and execution algorithms all reduce to an HJB equation. Optimal execution (how fast to sell a large block) and market-maker quoting (Avellaneda-Stoikov) are both HJB problems with different state variables.
• Solving HJB by hand only works for a few tidy cases (log or power utility, linear dynamics). Real trading desks solve it numerically — on a grid, or via reinforcement learning, which is secretly approximating the same value function.
• The verification theorem matters. A function that satisfies the HJB equation and the right boundary conditions is provably the optimal value — this is what justifies trusting a numerically computed policy in production.