Q-Learning

Suppose you want an agent to trade, route an order, or manage inventory well, but nobody can hand you a model of the market. You don't know the transition probabilities, you don't know what a given action does to prices, and you can't write down the dynamics. All you can do is act, watch what happens, and collect a number afterwards. Q-learning is the algorithm for exactly that situation: it learns how good each action is without ever learning how the world works.

Think of a commuter who has never seen a map. Every morning she picks a route, and every evening she remembers roughly how long it took. She isn't building a model of traffic flow; she's just keeping a running score for each choice at each junction. Over months, those scores get good enough that following the best-scoring turn at every junction produces a near-optimal commute. That scorecard is a Q-table, and the process of updating it after each trip is Q-learning.

The scorecard

The core object is a function $Q(s, a)$, read "the value of taking action $a$ while in state $s$". Here $s$ is the situation the agent finds itself in (holding 200 shares, spread is wide, ten minutes left), and $a$ is one of the choices available (buy, hold, sell). The number $Q(s,a)$ estimates the total future reward the agent should expect if it takes action $a$ now and then behaves optimally forever after.

If you had that function exactly, acting well would be trivial: look up every action and take the biggest number. All the difficulty is in getting the numbers.

The update rule

After each step the agent sees four things: the state it was in ($s$), what it did ($a$), the reward it got ($r$), and the state it landed in ($s'$). From those it forms a fresh estimate of $Q(s,a)$ and moves its old estimate a little way toward it:

$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right]$$

In plain English: the new score is the old score plus a fraction of your surprise. The bracket is the surprise, usually called the temporal-difference error. Inside it, $r + \gamma \max_{a'} Q(s',a')$ is what the reality of this step suggests the score should have been — the reward you actually banked, plus the best score available from wherever you ended up, shrunk by the discount factor $\gamma$ (between 0 and 1) because reward later is worth less than reward now. Subtracting the old score gives the gap. The learning rate $\alpha$ (a small number like 0.1) decides how much of that gap to close in one go.

Worked example: a single update

An agent is holding a position, so the state is long. Its current table says $Q(\text{long}, \text{hold}) = 2.00$ and $Q(\text{long}, \text{sell}) = 3.00$. It chooses hold, the position gains one unit of reward ($r = 1.0$), and it is still long afterwards. Use $\alpha = 0.5$ and $\gamma = 0.9$.

Step 1 — best value available from the new state:

$$\max_{a'} Q(\text{long}, a') = \max(2.00,\ 3.00) = 3.00 .$$

Step 2 — the target this step implies:

$$r + \gamma \max_{a'} Q(s',a') = 1.0 + 0.9 \times 3.00 = 3.70 .$$

Step 3 — the surprise, and the new entry:

$$Q(\text{long},\text{hold}) \leftarrow 2.00 + 0.5 \times (3.70 - 2.00) = 2.85 .$$

Only that one cell changed. Everything else in the table is untouched.

Worked example: why it converges

Repeat the same experience with the same numbers and watch the gap close. The target stays 3.70, and each update removes half the remaining distance:

update	old value	surprise	new value
1	2.000	1.700	2.850
2	2.850	0.850	3.275
3	3.275	0.425	3.488
4	3.488	0.212	3.594

The estimate walks toward 3.70 rather than jumping there. That is deliberate: real rewards are noisy, so a small $\alpha$ averages many samples instead of over-reacting to one lucky trade. With $\alpha = 0.5$ the gap halves every visit; with $\alpha = 0.1$ it shrinks by a tenth, slower but far steadier.

Exploration is not optional

The update only improves cells the agent actually visits. An agent that always takes its current best action will never test the alternatives, and a single unlucky early sample can bury a genuinely good action forever. The standard fix is epsilon-greedy: with probability $\epsilon$ (say $10\%$) pick a random action, otherwise pick the best-scoring one. Start $\epsilon$ high and decay it as the table firms up. See Exploration vs Exploitation.

The subtle and powerful part is that the $\max$ in the update refers to the best next action, not the action the agent will actually take next. So the agent can behave randomly while still learning the value of behaving optimally. That property is what makes Q-learning off-policy, and it is why you can train it on logged data generated by some other trader or execution algorithm.

Where it breaks in markets

A table needs one cell per state-action pair, which dies the moment the state includes anything continuous like a price or a spread. Replacing the table with a neural network gives Deep Q-Networks, at the cost of stability. And because the update takes a $\max$ over noisy estimates, Q-learning systematically overestimates values — the max of noisy numbers is biased upward. In a trading context that shows up as an agent that looks brilliant in training and mediocre live. Double Q-learning, which uses one estimate to pick the action and another to score it, is the usual remedy.

Q-learning is the workhorse entry point to value-based Temporal Difference Learning, and the direct ancestor of most of what gets deployed in Reinforcement Learning for Trading.