Multi-Agent Reinforcement Learning

Standard reinforcement learning imagines one agent acting in an environment that responds only to that agent's own actions — the environment might be complicated, but it isn't strategic. A market is nothing like that: when your execution algorithm places an order, other traders' algorithms react to it, and their reactions change the prices your algorithm sees next, which then changes what it does, which changes what they do. The environment isn't a fixed backdrop; it's made up of other agents who are themselves learning and adapting to you. Multi-agent reinforcement learning is the framework built for exactly this — multiple agents learning simultaneously in a shared environment where each one's actions are part of what the others experience as "the environment."

An analogy: a crowded intersection with no traffic lights

Picture drivers approaching a busy intersection with no traffic signals, each deciding independently whether to go or yield. A single-agent view would treat "the other cars" as a fixed, predictable pattern of obstacles to route around. But every other driver is also adapting in real time — if you notice cars usually yield to you at this intersection, you start going more aggressively, and if enough drivers adopt that same aggressive habit, the whole intersection's dynamic changes, and everyone has to relearn how to behave. The "right" strategy for any one driver depends on what all the other drivers are currently doing, which itself depends on what everyone did last week. That's the essence of multi-agent learning: the environment each agent faces is partly made of other agents who are learning too, so the ground keeps shifting under everyone at once.

What makes this different from single-agent RL

In single-agent RL, the environment's response to an action, while possibly complex or noisy, comes from a fixed (if unknown) process — practicing more against it converges toward an optimal policy. In multi-agent RL, the "environment" includes other learning agents whose policies are changing at the same time yours is, which means the problem you're solving today may look different tomorrow purely because everyone else adapted. This breaks the standard convergence guarantees of single-agent RL and introduces genuinely new questions: should agents cooperate or compete? Can they observe each other's actions, or only the aggregate effect (like the resulting price)? Is there a stable equilibrium where no agent wants to change its strategy given what the others are doing — analogous to a Nash equilibrium in game theory — or does the system cycle indefinitely as agents chase each other's adaptations?

Worked example: two market-making agents competing for order flow

Consider two simplified market-making agents quoting bid-ask spreads for the same instrument, each trying to capture order flow while managing inventory risk. If Agent A widens its spread to reduce risk, more order flow shifts to Agent B, who now captures more volume at Agent B's tighter spread — but Agent B also accumulates more inventory risk as a result. Agent B might respond by widening its own spread in turn once its inventory risk grows uncomfortable, shifting flow back toward Agent A. Trained independently with single-agent RL, each agent would treat the other's spread as a fixed feature of "the market" — but because both are adapting, the "optimal" spread for each one is a moving target that depends on what the other is currently doing, and a policy that looked great in isolation can perform very differently once both agents are learning at once.

What this means in practice

Multi-agent RL is the natural framework for modeling execution algorithms competing for the same liquidity, market makers reacting to each other's quotes, or any setting where a trading agent's own actions move the market that other participants (human or algorithmic) then react to. It's harder to train and validate than single-agent RL — convergence isn't guaranteed, results can be sensitive to how many agents are simulated and how they're allowed to observe each other, and a policy trained against static historical data may behave very differently once deployed against other adaptive agents live. Simulated multi-agent training environments are a common workaround, but they inherit whatever assumptions the simulator makes about how realistic agents actually behave.