Doubly Robust Off-Policy Estimators

Suppose you have historical data generated by an old execution policy (say, an order-routing rule) and want to estimate how a new proposed policy would have performed, without actually running it live. One approach models the outcome directly (predict the reward a state-action pair would produce) and simulates the new policy against that model; another reweights the historical data by how much more or less likely each action would be under the new policy versus the old one (importance sampling). Each approach alone is fragile: the outcome model is wrong if misspecified, and importance weights blow up in variance when the new and old policies disagree substantially.

A doubly robust estimator combines both: it starts from the outcome model's prediction and adds an importance-weighted correction term for the discrepancy between the model's prediction and the actually observed outcome. The key property is that the estimate stays statistically consistent (converges to the truth as data grows) if either the outcome model or the importance weights (the old policy's known action probabilities) are correctly specified — not both simultaneously.

This matters in practice because you rarely trust either component fully: the reward model might miss some nonlinearity, or the logged execution policy's probabilities might be only approximately known. Doubly robust estimation gives a second line of defense — one component compensating for error in the other — before evaluating a new policy on real capital.