Function Approximation in Reinforcement Learning

Basic reinforcement learning keeps a table: for every possible state, store a number saying how good that state is. That works fine for a board game with a few thousand positions, but a trading environment's state might include current price, recent returns, order-book depth, inventory, and time of day — a combination with effectively infinite distinct values. You cannot build a table with a row for every possible price; you'd never visit the same exact state twice, and a table with no entry for the state you're currently in is useless.

Function approximation replaces the table with a model — often a neural network — that takes a state as input and outputs its estimated value, the same way a real estate appraiser doesn't keep a lookup table of every house ever sold; they learn a function of features (square footage, location, condition) that generalizes to a house they've never specifically seen before, because it resembles houses they have. The appraiser doesn't need to have valued this exact house previously; they need to have learned a pattern that transfers.

The mechanics, one symbol at a time

Instead of a table entry $V(s)$ for each state $s$, function approximation learns a parametric function $\hat{V}(s; w)$ — for instance, a neural network with weights $w$ that takes a state's features and outputs an estimated value. Training adjusts $w$ to make $\hat{V}(s;w)$ match a target value $y$ (typically built from an observed reward plus a bootstrap estimate of the next state's value) using gradient descent on the squared error:

$$w \leftarrow w + \alpha \big(y - \hat{V}(s; w)\big) \nabla_w \hat{V}(s; w)$$

In words: compute the gap between the target $y$ (what the value "should" be, based on new experience) and the model's current estimate $\hat{V}(s;w)$, and nudge the weights $w$ in the direction that would have reduced that gap, scaled by learning rate $\alpha$ — the same weight-update rule as any neural network trained by gradient descent. The crucial difference from ordinary supervised learning is where $y$ comes from: it's often built partly from the model's own current estimate of a later state's value (this is called bootstrapping), which means the target the network is chasing keeps shifting as the network itself changes — a moving target the network is partly responsible for creating.

Worked example 1: one value update by hand

A simple linear function approximator, $\hat{V}(s;w) = w_1 \cdot \text{momentum} + w_2 \cdot \text{inventory}$, currently $w = (0.4, -0.2)$. Current state: momentum $=1.5$, inventory $=2.0$.

$$\hat{V}(s;w) = 0.4(1.5) + (-0.2)(2.0) = 0.6 - 0.4 = 0.2$$

Suppose the observed reward this step was $0.5$ and the estimated value of the next state is $0.6$, with discount factor $\gamma=0.9$: target $y = 0.5 + 0.9(0.6) = 0.5+0.54=1.04$.

Error: $y - \hat{V}(s;w) = 1.04 - 0.2 = 0.84$. With learning rate $\alpha=0.1$ and gradient $\nabla_w \hat{V} = (\text{momentum}, \text{inventory}) = (1.5, 2.0)$ for this linear model:

$$w_1 \leftarrow 0.4 + 0.1(0.84)(1.5) = 0.4 + 0.126 = 0.526, \quad w_2 \leftarrow -0.2 + 0.1(0.84)(2.0) = -0.2+0.168=-0.032$$

Both weights shift toward valuing momentum and inventory more highly this step, because the target suggested the current state is worth more than the model thought. The key point: this exact same weight update also silently changes the value estimate for every other state with similar momentum and inventory — that's generalization, and it's both the entire benefit and the entire risk of function approximation.

Worked example 2: generalization spilling over, for better and worse

Continue from above: the updated weights $(0.526, -0.032)$ now change the estimated value of a different state never directly visited this step, say momentum $=1.4$, inventory $=1.8$: $\hat{V} = 0.526(1.4) - 0.032(1.8) = 0.679$, versus $0.20$ before the update — more than tripled, purely as a side effect of an update triggered by a different state.

That's useful when nearby states genuinely have similar values — the agent doesn't need to visit every state individually to learn a good policy. But it's dangerous when the approximator's assumed shape doesn't match reality: if the true value function has a sharp discontinuity around inventory level 2.0 (say, a risk-limit breach), a linear model cannot represent that break and will systematically misvalue states on one side of it, no matter how much data it sees.

The explorer above shows crawling, converging, and overshooting on a fixed loss surface. Function approximation in RL is harder than that picture: because the target $y$ is built partly from the model's own shifting estimates, the "surface" itself can move as training proceeds, which is part of why RL training is often less stable than ordinary supervised training on a fixed dataset.

What this means in practice

Function approximation is what makes reinforcement learning applicable to realistic financial environments — execution algorithms, market making, portfolio allocation — where state spaces are continuous or effectively infinite. It's also the source of most of the field's notorious instability: bootstrapping (target built from the model's own output), off-policy learning (training on data from a different policy than the one being evaluated), and function approximation together are sometimes called the "deadly triad" — each is manageable alone, but the combination can make value estimates diverge rather than converge, even in simple toy problems. Techniques like target networks (freezing a slowly-updated copy of the model to compute targets, rather than chasing the live model's shifting output) and experience replay exist to blunt this instability.