Deep Q-Networks

Q-learning works by filling in a table: one row per state, one column per possible action, and each cell holding an estimate of how good that action is in that state. This works fine when the number of states is small enough to enumerate — a handful of market regimes, say. It breaks completely the moment the "state" is something like a raw order-book snapshot or a stretch of recent price history, because the number of distinct states is then effectively infinite, and there is no table large enough to hold a row for each one. A deep Q-network (DQN) solves this by replacing the table with a neural network: instead of looking up a stored value in a cell, you feed the state into the network and it predicts the value, generalizing to states it has never seen exactly before based on states that were similar.

Same target, different machinery

Ordinary Q-learning updates one table cell at a time toward a target built from the classic Q-learning equation: today's estimated value for a state-action pair moves toward the reward just received plus the best value achievable from the next state. A DQN keeps the same target but replaces the table lookup with a network's prediction, and trains the network's weights (via ordinary gradient descent) to make its output closer to that same target.

$$Q(s,a) \approx Q(s,a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s,a) \right]$$

In words: the current estimate for taking action $a$ in state $s$ is nudged, by a learning rate $\alpha$, toward the reward $r$ just observed plus the best possible value from the next state $s'$ (discounted by $\gamma$, which weighs future reward against immediate reward), minus the current estimate. In tabular Q-learning this nudges one cell; in a DQN, this same quantity becomes the error signal used to update the neural network's weights via backpropagation, so a single update can shift the network's predictions across many similar, never-before-seen states at once — the entire point of trading a table for a function approximator.

Worked sketch: why naive training was unstable, and the two fixes

Training a network this way turned out to be badly unstable, for a fixable reason: the "target" in the update above depends on the same network being updated (through $\max_{a'} Q(s',a')$), so every weight update shifts the target you were just chasing — a moving-goalpost problem that can make training diverge. The original DQN paper fixed this with two tricks. First, a separate, frozen target network — a copy of the weights, updated only occasionally — computes the target term, so the goalposts hold still between updates. Second, experience replay: transitions are stored in a buffer and sampled randomly for training instead of used in the order they occurred (consecutive market ticks are highly correlated), which makes each training batch closer to the independent, identically distributed data ordinary gradient-based training assumes.

The instability a naive DQN suffers from is visible in a picture like the one above as an overshooting, non-converging trajectory rather than a smooth descent — a moving target and correlated training batches are exactly the kind of thing that turns a well-behaved descent into an erratic one.

What this means in practice

DQNs make sense once the "state" genuinely cannot be discretized into a small table without losing information that matters — raw order-book depth, recent tick sequences, portfolio state across many assets at once. They are considerably harder to get working reliably than tabular Q-learning: more hyperparameters, more compute, and no convergence guarantee, so a simpler discretized state space that still captures what matters is often the more practical starting point.