Temporal Difference Learning

To learn what a state is worth, the obvious approach is to visit it, play the episode out, and average the totals. That is Monte Carlo estimation: unbiased, but painfully slow, and for a trading problem often impossible — if an "episode" is a full trading day, you get 252 lessons a year. Temporal difference learning refuses to wait.

The idea: revise as you go

You leave the office estimating 45 minutes to get home. Ten minutes in you hit traffic on a road that is normally clear. You do not suspend judgement until you park — you revise immediately, because the traffic itself is evidence.

That revision is the whole method. Your estimate of the trip changed because your estimate of the rest of the trip changed: you learned from a guess, before the truth arrived. TD does this arithmetically, treating the reward just received plus your current opinion of where you landed as a better estimate than your current opinion of where you left.

The update

For a state $S_t$ with current estimate $V(S_t)$, observing reward $R_{t+1}$ and next state $S_{t+1}$:

$$V(S_t) \leftarrow V(S_t) + \alpha \Big[ \underbrace{R_{t+1} + \gamma V(S_{t+1})}_{\text{target}} - V(S_t) \Big].$$

In words: form a target out of what you actually got plus a discounted guess about what comes next, see how far your old estimate was from it, and move a small fraction $\alpha$ of the way there. The bracketed quantity is the TD error, written $\delta_t$. It is the surprise — positive means things went better than the current value function predicted.

Notice the target is just the right-hand side of the Bellman equation with the expectation replaced by a single sample. That is the entire relationship between the two ideas.

Worked example: two updates in one episode

Take $\alpha = 0.1$ and $\gamma = 0.9$. Current estimates are $V(A) = 5.00$, $V(B) = 6.00$, $V(C) = 4.00$.

Step one. From $A$ you receive a reward of $1$ and land in $B$. The target is $1 + 0.9(6.00) = 6.40$, so the TD error is $6.40 - 5.00 = 1.40$ and

$$V(A) \leftarrow 5.00 + 0.1(1.40) = 5.14 .$$

Step two, immediately after. From $B$ you receive $2$ and land in $C$. The target is $2 + 0.9(4.00) = 5.60$, the error is $5.60 - 6.00 = -0.40$, and

$$V(B) \leftarrow 6.00 + 0.1(-0.40) = 5.96 .$$

Two states improved before the episode was anywhere near over. A Monte Carlo learner would still be waiting, holding no information at all.

Worked example: why the variance is lower

State $A$ always leads to $B$ with zero reward; from $B$ the episode ends with $+10$ or $-4$, equally likely. Take $\gamma = 1$, so the true value of $A$ is $3$. Start with $V(A) = 5$ and suppose $V(B) = 3$ is already learned.

Monte Carlo, using the realised episode return:

• Episode 1 returns $+10$: $\;V(A) = 5 + 0.1(10 - 5) = 5.50$.
• Episode 2 returns $-4$: $\;V(A) = 5.50 + 0.1(-4 - 5.50) = 4.55$.

TD, using the target $0 + V(B) = 3$ both times:

• Episode 1: $\;V(A) = 5 + 0.1(3 - 5) = 4.80$.
• Episode 2: $\;V(A) = 4.80 + 0.1(3 - 4.80) = 4.62$.

Both head toward $3$, but look at the path. The Monte Carlo updates carry the full episode noise — a standard deviation of $7$, so each one jerks $A$ by about $0.7$ in a random direction. TD's updates carry none of it: the coin flip lands entirely on $B$, where it belongs. TD trades a little bias, from bootstrapping off an imperfect $V(B)$, for a large cut in variance. On noisy financial data that trade is almost always worth making.

Running averages settle on the truth at roughly one over the square root of the sample count, and TD estimates behave the same way. Drag the sample count below and note how much of the early wobble is pure sampling noise rather than signal.

In practice

The step size $\alpha$ is the main dial. Large values track a changing world quickly but leave the estimate rattling; small values are stable but slow to notice a regime change. For a stationary problem $\alpha$ should decay; for markets a small constant $\alpha$ is usually preferred, precisely because it lets the estimate forget.

TD is also the engine under the algorithms you will actually deploy. Q-learning is TD on action values with a max in the target; SARSA is the same with the action you actually took; actor-critic methods use the TD error directly as the signal telling the policy whether a move beat expectations.