Sanity Checks for Saliency Metrics

By Richard J. Tomsett, Daniel Harborne, Supriyo Chakraborty, Prudhvi K. Gurram, Alun David Preece. 2019

210 citations recorded by the source corpus.

Saliency maps are a popular approach to creating post-hoc explanations of image classifier outputs. These methods produce estimates of the relevance of each pixel to the classification output score, which can be displayed as a saliency map that highlights important pixels. Despite a proliferation of such methods, little effort has been made to quantify how good these saliency maps are at capturing the true relevance of the pixels to the classifier output (i.e. their “fidelity”). We therefore investigate existing metrics for evaluating the fidelity of saliency methods (i.e. saliency metrics). We find that there is little consistency in the literature in how such metrics are calculated, and show that such inconsistencies can have a significant effect on the measured fidelity. Further, we apply measures of reliability developed in the psychometric testing literature to assess the consistency of saliency metrics when applied to individual saliency maps. Our results show that saliency metrics can be statistically unreliable and inconsistent, indicating that comparative rankings between saliency methods generated using such metrics can be untrustworthy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

Citations in the corpus

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “39 Indeed, much current research into producing ‘‘post hoc’’ explanations 40 of (uninterpretable) neural network outputs has resulted in techniques that are difﬁcult to validate, 41 with some failing basic sanity checks.”
Debugging Tests for Model Explanations (2020, background): “However, Tomsett et al. [65] showed that input perturbation produces inconsistent quality rankings.”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “We measure the ﬁdelity [203] of the attributions by successively removing the features that have the highest mean importance assigned (most relevant ﬁrst (MoRF) [203]).”