Calibrating a Simulator Against Live Fills

A simulator estimates that a resting limit order posted one tick from the touch fills 70% of the time within thirty seconds — a number derived carefully from queue-position modeling, historical trade volume, and an assumed cancellation rate for orders ahead of it. It's a reasonable model, built the way the textbooks describe. Deployed live, the actual fill rate at that same distance from the touch is 44%. The model wasn't wrong in its structure — it correctly identifies that fill probability depends on queue depletion — it was wrong in one of its inputs: the assumed cancellation rate for orders ahead of the queue was too low, because in this particular name, a large share of resting size belongs to algorithms that cancel and replace on every tick, churning the queue in a way the simulator's static assumption never captured.

No amount of getting the mechanics conceptually right saves a simulator whose parameters are wrong, and the only way to know they're wrong is to check the simulator's predictions against something it didn't have access to when it made them — namely, what actually happened when real orders were sent to the real market. That comparison, and the process of adjusting the simulator's parameters until the gap closes, is calibration.

What gets compared, and how

Send real orders — starting small, since the point is to learn the market's behavior cheaply, not to prove the strategy at scale yet — and log everything the simulator would need to reproduce that exact situation: the queue depth ahead of the order at the moment it joined, the market conditions over the following window, and whether and when it actually filled. Feed the identical situation into the simulator and compare its predicted fill probability, or its predicted fill price, to what really happened. Do this across enough orders, and across different market conditions (open versus midday, calm versus volatile), to get a distribution rather than a single lucky or unlucky comparison.

The gap between prediction and reality points at which parameter to adjust: a fill rate that's too high across the board usually means the assumed cancellation rate ahead of your order is too low, or vice versa; a fill price that's systematically worse than predicted often means the impact-per-share assumption is too small. Adjust the relevant parameter, re-run the comparison on fresh live data, and repeat — calibration is a loop, not a one-time fit.

Worked example: closing a fill-rate gap in three rounds

Round	Assumed cancel rate (ahead-queue, per minute)	Predicted fill rate	Observed fill rate	Gap
1 (initial model)	8%	70%	44%	26 pts
2 (recalibrated)	22%	51%	46%	5 pts
3 (recalibrated again)	26%	47%	45%	2 pts

Each round uses the previous round's observed data to fit a new cancellation-rate assumption, then checks that new assumption against a fresh batch of live orders — not the same batch used to fit it, since checking a model against the data used to build it would just confirm the fit rather than test it. By round 3 the model's prediction is within noise of what actually happens, and the earlier 26-point gap — large enough to invalidate any capacity or Sharpe estimate that depended on it — is gone.

Doing it properly

Start the live sampling small and expand only once the simulator's predictions track reality, since the point of calibration is to learn cheaply, not to prove the strategy at scale before it's trustworthy. Recalibrate on a schedule rather than once at launch — a queue's typical composition (how much of it churns versus sits still) changes with the participants trading a name, so a calibration that was accurate six months ago can drift stale exactly like the signal itself, see Monitoring Signal Decay in Production. And keep the comparison granular — by time of day, by name, by market regime — because a parameter that's well calibrated on average can still be badly wrong in the specific slice where you actually trade the most size.

Be explicit about which parameters you're calibrating and hold everything else fixed while you do, since adjusting several at once makes it impossible to tell which change actually closed the gap and which one happened to cancel another out. Where possible, calibrate against a metric closer to what you ultimately care about than raw fill rate alone — realized slippage, or the strategy's net PnL under the recalibrated assumptions — since a model can match fill rate on average while still getting the distribution of fill prices wrong in a way that only shows up once it's translated into dollars. Finally, treat a calibration that never needs adjusting as a reason for suspicion rather than comfort: a simulator whose gap to live fills stays exactly zero for months is more likely being checked against too narrow a slice of conditions than genuinely capturing every regime the strategy will eventually trade through.