Explanation Stability and Robustness

If two inputs are nearly identical and a model gives them nearly identical predictions, a reasonable person would expect any honest explanation of those predictions to also be nearly identical — the reasoning behind two almost-the-same decisions shouldn't look wildly different. That expectation is called stability (or robustness), and it turns out to fail badly for several widely used attribution methods, including some gradient-based saliency methods and, under certain conditions, LIME and SHAP. It's possible to construct — or simply stumble into — a tiny perturbation to an input, one small enough that the model's prediction barely moves at all, that completely reorders which features the attribution method claims were most important. The explanation changes dramatically while the thing it's supposedly explaining barely changes at all.

This is a distinct failure from unfaithfulness. An unfaithful explanation gets the ranking wrong for a given input. An unstable explanation can be locally faithful at every single point and still swing wildly between two points that are almost the same input, which is arguably worse for trust: it means the explanation you get depends heavily on which exact version of an input you happened to feed the method, not on anything robust about the model's actual behavior.

Worked example

A quant explains a credit-risk model's decision for one applicant using LIME, getting a ranking that puts debt-to-income first, account age second, and recent-inquiry count a distant third. A colleague, testing the method's robustness, perturbs the applicant's profile by an amount within normal data-entry noise — nudging one minor feature (say, reported monthly rent) by a few percent, small enough that the model's predicted default probability moves from 24% to 24.3%, an essentially unchanged decision. Recomputing the LIME explanation on this barely-different input, the ranking comes back completely reordered: recent-inquiry count now ranks first, debt-to-income drops to third. Two applicants who are, for all practical purposes, the same person would receive two completely different stated "reasons" for their nearly identical risk scores — a serious problem if either explanation is ever used to justify the decision to the applicant or a regulator.

input B (tiny nudge) → prediction 24.3% 1. inquiries 2. account age 3. debt/income

predictions nearly identical, rankings completely reversed

What this means in practice

Stability matters most exactly where explanations are used to justify decisions to someone outside the modeling team — credit denials, trade-surveillance flags, model-override justifications in regulated finance — because an unstable explanation invites the follow-up question "would a nearly identical case have gotten a completely different stated reason?" and the honest answer, for many methods, is yes. Practical mitigations include averaging attributions over several small random perturbations of the input (smoothing out the instability) and explicitly reporting how much a given explanation moves under small input perturbations, alongside the explanation itself.