Measuring Explanation Faithfulness

An attribution method can pass the basic sanity checks — it responds to the actual trained model, not just to the raw input — and still be a poor description of what the model is really doing. Faithfulness asks a sharper, more useful question: does the ranking of features an explanation gives actually match the features the model is really relying on to make its prediction? An unfaithful explanation might correctly notice that the model has learned something, while still getting the relative importance of individual features wrong.

The standard way to measure faithfulness quantitatively is a deletion (or insertion) test. Take an explanation's ranking of features from most to least important for a given prediction, then progressively remove (mask, zero out, or replace with a baseline value) the top-ranked features one at a time, recomputing the model's prediction after each removal. If the explanation is faithful, removing the features it claims matter most should cause the prediction to degrade — move away from its original value — the fastest; removing unimportant features first should barely move the prediction at all. Plotting prediction accuracy or confidence against the fraction of top-ranked features removed traces a curve, and the area under that curve (or the drop after removing just the first feature or two) gives a single number comparing how faithful different explanation methods are for the same model.

Worked example

A quant compares two attribution methods — SHAP and a simple gradient-based method — explaining the same prediction from a volatility-regime classifier, using six input features. Both methods rank the features, but in a different order; SHAP puts realized volatility first, the gradient method puts a lagged return term first. Running the deletion test: masking SHAP's top-ranked feature (realized volatility) and recomputing drops the model's confidence in its original prediction from 92% to 58% — a big move, consistent with that feature genuinely driving the prediction. Masking the gradient method's top-ranked feature (the lagged return) only drops confidence from 92% to 87% — a much smaller move, even though the gradient method claimed it was the most important feature. That gap is direct, quantitative evidence that SHAP's ranking is more faithful to what this particular model is actually doing for this prediction, not just a matter of which method "looks" more reasonable.

What this means in practice

Faithfulness testing turns "which explanation method should we trust here" from a matter of taste into something measurable, model by model and prediction by prediction — a method can be faithful for a tree ensemble and less faithful for a neural network on the same data, so the check is worth rerunning whenever the underlying model changes materially. It's particularly worth running before an explanation method is used to justify a real decision, like overriding a trading signal because "the model relied on a spurious feature" — that claim should survive a deletion test before it's acted on.