Sanity Checks for Saliency Methods

A saliency map — a heatmap showing which parts of an input a model supposedly "focused on" to make a prediction — is visually persuasive almost by default: any highlighted region of a chart, image, or feature vector looks like it's telling you something. That persuasiveness turned out to be a trap. A well-known study found that several popular saliency methods produce heatmaps that look nearly identical whether the underlying model has been trained normally, or has had its weights completely randomized and never trained at all, or has been trained on completely scrambled, meaningless labels. If an explanation method still points to "the same important regions" after the model it's supposedly explaining has been reduced to noise, the method was never actually reading the model's learned reasoning — it was picking up something else entirely, often just edge-like structure already present in the raw input.

The proposed fix is a family of sanity checks, run before trusting any attribution method on a real problem. The model-randomization test: replace the trained model's weights with random ones, layer by layer, and check whether the explanation changes meaningfully — it should, since a randomly-weighted model has learned nothing and any faithful explanation of it should look like noise, not like a crisp, structured heatmap. The data-randomization test: train the model on the same inputs but with labels shuffled at random, then check that the explanation for the resulting (necessarily meaningless) model differs sharply from the explanation of a properly trained model. A method that passes both is at least responsive to what the model actually learned; a method that fails either one is producing something closer to a fixed function of the raw input, dressed up as an explanation.

Worked example

A quant applies a saliency-style attribution method to a neural network that predicts next-bar direction from a short window of recent price-and-volume data, visualized as a heatmap over the input window. The heatmap for the trained model highlights a spike in volume three bars back. As a sanity check, the team re-randomizes the final layer's weights, leaving the model no better than random guessing, and recomputes the same attribution — if the method is sound, this heatmap should look meaningfully different (structureless, since there's no learned signal left to attribute). In this case it comes back almost pixel-for-pixel identical to the original, still highlighting the same volume spike. That's the sanity check failing: the method was responding to something about the raw input's structure (the volume spike being a locally large value) rather than to anything the trained model had learned to use, and the team discards that particular attribution method rather than trusting its output on real predictions.

weights randomized heatmap: identical highlight

same heatmap despite a model that has learned nothing → sanity check fails

What this means in practice

Sanity checks are a cheap, mechanical filter to run before adopting any attribution method for a real monitoring or explanation pipeline — a few minutes of randomizing weights and shuffling labels can rule out methods that would otherwise produce confident-looking but empty heatmaps in production. Passing the sanity check is necessary but not sufficient for an explanation to be trustworthy; it rules out the worst failure mode (not responding to the model at all) without guaranteeing the explanation is faithful to the model's real reasoning in every case.