Experience Replay

Training a neural network on a stream of consecutive experiences is a bad idea — consecutive states in a trading simulation or a game look almost identical to each other, so the network overfits to whatever streak it is currently on and forgets what it learned a few minutes earlier. Experience replay fixes this by storing every transition — state, action, reward, next state — in a large buffer and training on random samples drawn from across that buffer instead of on the most recent transition alone.

Two benefits fall out of this. First, decorrelation: a random minibatch mixes old and new experiences, so gradient updates are not dominated by whatever the agent happens to be doing right now. Second, sample efficiency: a single transition can be replayed and learned from many times rather than being thrown away after one use, which matters when generating new experience — running a real trading strategy live, say — is expensive.

Worked example. A buffer holds the last 1,000,000 transitions. Instead of training on the newest transition after each step, the agent draws a random minibatch of 32 transitions from anywhere in that buffer, computes the TD error for each, and updates the network — repeating this many times per new transition collected, squeezing far more learning out of each real interaction.

The buffer has a fixed size and evicts the oldest entries first, so it gradually shifts to reflect the agent's current policy rather than staying frozen on ancient, no-longer-relevant behavior.