Model-Based RL and World Models

Q-learning, REINFORCE, and actor-critic methods are all model-free: they learn purely from lived experience — try an action, observe a reward, update an estimate — without ever building an explicit representation of how the world responds to actions in general. Model-based reinforcement learning takes the opposite approach: it first learns an explicit model — given this state and this action, what state and reward come next — and then uses that learned model to plan, mentally simulating many possible futures before ever acting in the real world again. The trade-off is straightforward: real trials in markets are expensive and risky, and a decent model lets you generate cheap, simulated trials instead.

The world model, concretely

A world model is, at minimum, a function predicting the next state and reward given the current state and action — trained the ordinary supervised-learning way, on observed (state, action, next-state, reward) transitions, exactly the kind of data a model-free method would also collect but here used for a different purpose. Once this model exists, an agent can generate rollouts entirely in its own head: starting from a state, simulate an action using the model's predicted next state, simulate another action from there, and so on, accumulating a simulated trajectory without ever touching the real market. A policy or value function can then be trained (or improved) on a mix of real and simulated experience, effectively multiplying a limited supply of real trading days into a much larger supply of plausible synthetic ones.

Worked sketch: ten real days versus ten thousand simulated ones

Suppose a firm has ten years of daily data for a market — a genuinely limited supply of real experience, since markets do not repeat identical days. A model-free agent trained only on those ten years has seen exactly that many real transitions, no more. A model-based approach instead fits a world model — say, next-day return distribution conditional on today's volatility and trend — using that same data, then uses the fitted model to generate thousands of plausible simulated day-sequences. A policy trained on this larger simulated dataset can genuinely benefit if the world model approximates reality well — but if the model learned the wrong dynamics (say, it underestimates how volatility clusters after a shock), the policy will confidently learn to handle a world that does not exist, and that confidence will not survive contact with real markets.

Simulated rollouts from a world model look exactly like the sampled paths above: many plausible trajectories generated from an underlying assumed process. The entire value of model-based RL rests on that underlying process actually resembling the real market closely enough — a world model built on a wrong assumption generates paths that look just as smooth and plausible as ones built on a correct one, with nothing in the picture itself to flag the difference.

What this means in practice

Model-based approaches are most valuable exactly where model-free methods struggle most: settings where real trials are scarce, slow, or expensive, which describes most trading contexts far better than it describes, say, a video game that can be replayed millions of times cheaply. The catch is that fitting an accurate model of financial dynamics is itself a hard, unsolved problem — markets are noisy and nonstationary — so model-based RL in finance inherits both the reinforcement-learning problem and the (also hard) forecasting problem, and errors in the second contaminate the first.