Counterfactual Risk Minimization

Suppose a system already logs which action it took for each situation and what reward followed (which ad it showed, which trade size it picked) but you want to train a better policy without running costly new experiments. You can't just train on the logged data as if it were ordinary supervised learning, because the old policy only ever tried some actions in each situation, never all of them, so naively optimizing on what happened overweights whatever the old policy already favored.

Counterfactual risk minimization corrects for this using importance weighting: each logged outcome is re-weighted by how likely the new candidate policy would have been to take that same action, divided by how likely the old logging policy was to take it. This produces an unbiased estimate of how well a new policy would have performed, even though it was never actually run — but the technique adds a variance penalty on top of the reweighted average, because importance weights can blow up when the new policy wants to take actions the old policy rarely tried.

Worked example

An ad platform logged impressions where an old policy showed ad A 90% of the time and ad B only 10% of the time. A new candidate policy that strongly prefers ad B would need to importance-weight the rare ad-B outcomes by roughly 9x to correct for how underrepresented they were in the logs. Counterfactual risk minimization keeps this reweighted estimate but penalizes it for the high variance those large weights introduce, discouraging the optimizer from picking a policy that looks great only because a handful of rare, heavily-upweighted log entries happened to have good outcomes.