RL Market-Making Agents

A market maker's core dilemma is simple to state and hard to solve well: quote a spread too wide and you never get filled; quote it too tight and you get filled constantly, including by informed traders, accumulating exactly the positions you least want to hold right before the price moves against you. The classical approach solves a stylized version of this tradeoff in closed form (see Stochastic Control Formulations Of Market Making), elegant but reliant on assumptions — constant order-arrival rates, simple price dynamics — that real markets routinely violate.

RL market-making agents instead learn a quoting policy directly from experience (simulated or historical), letting the agent discover how to adjust its bid and ask quotes in response to its own inventory and to observed market conditions, without requiring the closed-form assumptions a classical model needs — at the cost of needing a great deal of training data or a well-calibrated simulator to learn from.

What the agent controls and learns

A typical RL market-making setup has the agent choose, at each timestep, quote parameters such as:

• Bid and ask offsets from a reference price — how far inside or outside the current mid to place each quote.
• Quote sizes — how many shares/contracts to offer at each side.

conditioned on a state that includes the agent's own current inventory (a market maker running a large long position should skew its quotes to encourage selling and discourage further buying), recent volatility, and order-flow imbalance. The reward is typically the spread captured on each fill, minus a penalty on inventory risk — directly analogous to the risk-adjusted reward functions used in execution RL (see PnL vs Risk-Adjusted Reward Functions), since an agent rewarded purely on realized spread has no incentive to manage inventory risk at all.

Worked example

An RL market-making agent is trained on simulated order flow with a reward equal to spread captured per fill, minus $0.02 \times \text{inventory}^2$ as a quadratic inventory penalty. With zero inventory, it learns to quote a symmetric spread of 4 ticks around the reference price. After a run of buy fills pushes inventory to +500 shares (long), the learned policy skews asymmetrically: it widens the bid to 6 ticks below reference while narrowing the ask to 1 tick above — actively encouraging fills that bring the position back toward zero. Against a naive symmetric baseline that never adjusts for inventory, this skewing policy cuts the standard deviation of end-of-day inventory by roughly 60%, at a small cost to average captured spread — exactly the tradeoff the quadratic penalty was designed to produce.

What this means in practice

RL market-making is attractive precisely where classical stochastic-control solutions become intractable — many correlated instruments, complex order-flow dynamics, venues where the theoretical assumptions don't hold — but it inherits every weakness of RL generally: it needs either a large amount of real trading data or a simulator realistic enough that the learned policy transfers to live markets.