Robustness and Adversarial Evaluation

A model scores 96% accuracy on a clean, carefully assembled test set and gets deployed. In production, a data feed occasionally arrives a few seconds late, a price gets reported with one extra decimal place from a vendor glitch, or — in a market-making context — other participants notice the model's behavior and start feeding it inputs designed specifically to trigger a bad quote. None of this shows up in standard test-set accuracy, which only measures performance on inputs that look exactly like the clean historical data the model was validated on. Robustness and adversarial evaluation deliberately test the model against inputs that are realistically imperfect or actively adversarial, rather than only the pristine inputs it was built and tested on.

Two different questions, one evaluation family

Robustness testing asks: does the model's output change wildly under small, non-malicious perturbations that could plausibly occur — a stale price, a missing feature filled with a default, a slightly delayed timestamp? Adversarial evaluation goes further and asks: if a rational, self-interested counterparty knew how this model worked, could they construct an input specifically designed to fool it? Both are measured the same way: perturb the input by some small amount $\delta$ and check how much the output changes.

$$\text{Sensitivity} = \frac{\|f(x+\delta) - f(x)\|}{\|\delta\|}$$

In plain English: how much does the model's prediction move per unit of input perturbation — a model with low sensitivity to realistic noise is robust; a model where a tiny, carefully chosen $\delta$ flips the prediction entirely is fragile and, if that fragility is exploitable, a target for gaming.

Worked example: stress-testing a quote model

A market-making model sets bid-ask quotes based on five inputs: last trade price, order book imbalance, recent volatility, inventory, and time since last trade. A robustness test perturbs each input one at a time by a realistic amount — inventory off by one lot, volatility estimate stale by 30 seconds, order book imbalance recomputed after a single added order — and checks the resulting quote shift. Four of the five inputs shift the quote by less than half a tick under realistic perturbation; the order book imbalance input, however, shifts the quote by a full tick from a single small resting order being added and removed. That's a concrete, actionable finding: a savvy counterparty could place and cancel small orders specifically to walk the quote in a favorable direction, a vulnerability invisible from the model's clean backtest accuracy but obvious once someone tested sensitivity input by input.

The explorer above shows how a model's parameters move step by step on a loss surface — the same intuition applies to adversarial evaluation in reverse: an adversary is effectively searching for the smallest input change that pushes the model's output the furthest, a similar hill-climbing process aimed at the model's weak points instead of its training loss.

What this means in practice

Before deploying any model whose inputs another market participant could plausibly influence — a quoting engine, a lending decision an applicant can game by adjusting reported figures, a fraud model an attacker studies — run perturbation tests on each input individually and in combination, looking specifically for inputs where a small, cheap change produces a disproportionately large output swing. Where such a vulnerability is found, the fix is usually to add smoothing, sanity bounds, or an ensemble that doesn't rely on any single fragile input, rather than assuming the clean-data test-set accuracy already covers this risk.