Learning Quoting Policies for Market Making

A supervised model can be trained to predict where the price is going next. A market maker doesn't just predict — every quote it posts changes the world it's predicting in. Quote a wide, defensive spread and you trade less often but keep the trades you get; quote a tight, aggressive spread and you trade more often, accumulating inventory that then has its own risk. The next moment's optimal quote depends on the inventory the previous quote left behind. This is a sequential decision problem with feedback, not a one-shot prediction problem, and it's why market making has become one of the natural homes for reinforcement learning inside quant trading, alongside more classical closed-form approaches like Avellaneda-Stoikov.

Why this isn't ordinary supervised learning

Supervised learning trains on (feature, outcome) pairs where the model's own prediction doesn't change the outcome. A quoting policy trained the same way — "given this book state, what spread would have been most profitable" — evaluated against historical fills, has a subtle problem: those historical fills happened under whatever quoting policy actually generated them. A different, more aggressive policy would have traded with different counterparties, at different sizes, changing the inventory path and therefore every subsequent decision. This is the same causal-inference problem that own-fills impact estimation runs into (see Learning Market Impact From Your Own Fills), but compounded, because now the actions themselves — not just their side effects — are what's under evaluation.

Reinforcement learning frames this properly: the policy is a function from state (inventory, recent volatility, order book shape, time to close) to action (bid and ask quotes, or the skew and width around a reference price), and it's trained to maximise cumulative reward — typically P&L net of inventory risk — over a simulated or replayed environment where the consequences of different actions can actually be modelled, not just read off from history that assumes one fixed policy.

A worked example

A team trains a reinforcement-learning quoting policy on simulated order flow: a stochastic model of arrival rates for market orders that depend on the posted spread (wider spread, fewer fills, consistent with the closed-form Avellaneda-Stoikov intuition), plus an inventory penalty in the reward function that grows quadratically with position size, discouraging the policy from accumulating large one-sided inventory.

In simulation, the learned policy converges to a sensible-looking behaviour: quote tighter and more symmetric when inventory is near zero, skew the quotes to favour trades that reduce inventory as position grows, and widen defensively when recent volatility spikes — qualitatively similar to what the closed-form inventory-skew formula would prescribe, which is a useful sanity check that the learned policy hasn't found some degenerate way to exploit the simulator instead of solving the real problem.

The team then tests the same policy against a held-out slice of real historical data using a fill simulator that estimates, for a hypothetical quote the policy wouldn't have actually posted historically, whether and when it would plausibly have been filled (see Fill Simulator Fidelity for Training Execution Models). Performance under this more realistic counterfactual evaluation is noticeably worse than in the idealised simulator — a common and expected result, and the size of that gap is itself the most useful number the team gets, since it bounds how much to trust the simulator-only training environment before risking capital.

This quadratic shape is a stand-in for the inventory penalty in the reward function — small positions cost little, but the penalty accelerates as inventory grows. Adjust the curve steeper and the learned policy should skew its quotes more aggressively to shed inventory sooner; too gentle a penalty and the policy will tolerate large, risky positions in exchange for slightly better spread capture.

Where it breaks in practice

Simulators are the whole game here, and they are also the easiest place to be fooled: a simulator whose fill probabilities don't correctly capture adverse selection — the fact that aggressive counterparties disproportionately hit stale or mispriced quotes — will train a policy that looks profitable in simulation and loses money live, because it has learned to exploit an unrealistically benign counterparty model rather than a realistic one.