Reinforcement Learning

Read these first

1. 01Framing a Trading Problem as an MDP

   Before any reinforcement-learning algorithm can touch a trading problem, someone has to decide what the state, action and reward actually are, and that framing decides more of the outcome than the algorithm does.

2. 02Market Simulators for RL Training

   A reinforcement-learning agent needs to trade millions of times to learn, far more than any live market can safely provide, so it trains against a simulator, and the simulator's realism is a ceiling on everything downstream.

3. 03Markov Decision Processes

   The standard way to write down a problem where your action changes the situation you face next. Five ingredients - states, actions, transitions, rewards and a discount - turn "what should I do?" into something a computer can actually optimise.

4. 04Offline RL on Historical Market Data

   Offline reinforcement learning trains a policy from a fixed batch of past trades without ever letting the agent explore live, solving the safety problem of online RL by trading it for a harder statistical one.

5. 05Q-Learning

   A way to learn the value of every action in every situation purely from experience, without ever being told how the world works. Q-learning keeps a scorecard of action values and nudges each entry toward what actually happened.

6. 06Reward Design for Trading Agents

   A reinforcement-learning agent optimizes exactly the number you give it, not the number you meant, and in trading, those two numbers diverge in specific, predictable, expensive ways.

7. 07Temporal Difference Learning

   Learn from a single step instead of waiting for the episode to finish. TD nudges your estimate of a state's value toward the reward you just saw plus your current guess about where you landed - learning from a guess, using a guess.

8. 08The Bellman Equations

   The trick that makes sequential decision problems solvable - split the value of being somewhere into "what I get now" plus "the discounted value of wherever I end up next". One line of self-reference turns an infinite sum into a system you can actually solve.

9. 09The Sim-to-Real Gap in Trading RL

   A policy trained to perform beautifully in simulation can lose money the day it goes live, because it learned to exploit regularities of the simulator that the real market doesn't share.

10. 10The Policy Gradient Theorem

    A recipe for improving a decision-making policy by nudging up the probability of actions that turned out well, even when the environment is a black box you cannot differentiate through. It is the foundation of REINFORCE, actor-critic and PPO.

Then the rest

1. 11Exploration vs Exploitation
2. 12Action Spaces for Order Placement
3. 13Benchmarking RL Execution Agents Against TWAP
4. 14Contextual Bandits
5. 15Modelling Market Response to Agent Orders
6. 16Multi-Agent Reinforcement Learning
7. 17Multi-Armed Bandits
8. 18On-Policy vs Off-Policy Learning
9. 19PnL vs Risk-Adjusted Reward Functions
10. 20Proximal Policy Optimization
11. 21Putting Transaction Costs Inside the Reward
12. 22Reward Shaping and Reward Hacking
13. 23Risk-Sensitive Reinforcement Learning
14. 24RL for Portfolio Allocation
15. 25RL Market-Making Agents
16. 26State Design for Trading Agents
17. 27Thompson Sampling
18. 28Value Iteration and Policy Iteration
19. 29Actor-Critic Methods
20. 30Constrained RL Under Risk Limits
21. 31Deep Q-Networks
22. 32Function Approximation in Reinforcement Learning
23. 33Hierarchical RL for Multi-Timescale Trading
24. 34Model-Based RL and World Models
25. 35Off-Policy Evaluation of Trading Policies
26. 36Offline Reinforcement Learning
27. 37Partially Observable MDPs
28. 38The REINFORCE Algorithm