Interpretation of Neural Networks is Fragile

By Amirata Ghorbani, Abubakar Abid, James Y. Zou. 2017

1,049 citations recorded by the source corpus.

In order for machine learning to be trusted in many applications, it is critical to be able to reliably explain why the machine learning algorithm makes certain predictions. For this reason, a variety of methods have been developed recently to interpret neural network predictions by providing, for example, feature importance maps. For both scientific robustness and security reasons, it is important to know to what extent can the interpretations be altered by small systematic perturbations to the input data, which might be generated by adversaries or by measurement biases. In this paper, we demonstrate how to generate adversarial perturbations that produce perceptively indistinguishable inputs that are assigned the same predicted label, yet have very different interpretations. We systematically characterize the robustness of interpretations generated by several widely-used feature importance interpretation methods (feature importance maps, integrated gradients, and DeepLIFT) on ImageNet and CIFAR-10. In all cases, our experiments show that systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly susceptible to adversarial attack. Our analysis of the geometry of the Hessian matrix gives insight on why robustness is a general challenge to current interpretation approaches.

Subfields: Artificial Intelligence, Materials Chemistry, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Robustness Analysis Robustness of post-hoc ex-tractive interpretability methods has been studied (Kindermans et al., 2019; Ghorbani et al., 2019; Heo et al., 2019; Zheng et al., 2019; Slack et al., 2020).”
Foundation Models and Fair Use (2023, background): “…from difficulties in scaling due to high computational cost (e.g., leave-k-out retraining can be costly) (Feldman & Zhang, 2020; Zhang et al., 2021) or being inaccurate or erroneous when applied to complex but realistic model classes (Basu et al., 2020; Ghorbani et al., 2019; Søgaard et al., 2021).”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “[103] shows that an image can be generated that is perceptibly indistinguishable from the original but produces radically different interpretation.”
Sanity Checks for Saliency Maps (2018, background): “[23] Amirata Ghorbani, Abubakar Abid, and James Zou.”
Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values (2018, methodology): “To quantitatively assess the similarity between two explanations, for a given sample, we use the Spearman’s rank order correlation metric inspired by the work of Ghorbani et al. (2017).”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, background): “Saliency maps may also be vulnerable to adversarial attacks (Ghorbani et al., 2017).”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “There are several other challenges associated with post hoc explanations which are not captured by the fidelity metric: (i) they may fail to capture causal relationships between input features and black box predictions [15, 19], (ii) there could be multiple high-fidelity explanations for the same bl”
Explainable machine learning in deployment (2019, background): “tion between adversarial robustness and model interpretations [18, 21, 23, 59, 69].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, background): “[26] suggest that relying on visual assessment is not adequate and can be misleading.”
Is Attention Interpretable? (2019, background): “In this work, we focus on whether the attention weights’ suggested importance ranking of I faithfully describes why the model produced its output, echoing existing work on explanation brittleness for other model components (Ghorbani et al., 2017).”
Pathologies of Neural Models Make Interpretations Difficult (2018, background): “Our work is closely related to recent explorations into failures of interpretation methods (Kindermans et al., 2017; Ghorbani et al., 2017).”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, neutral): “, 2019), or others (Ghorbani et al., 2019; Feng et al., 2018).”
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods (2019, background): “[8] show that some explanation techniques can be highly sensitive to small perturbations in the input even though the underlying classifier’s predictions remain unchanged.”
NILE : Natural Language Inference with Faithful Natural Language Explanations (2020, background): “Difﬁculty in building interpretation models and the lack of robustness of the same are some of the major issues in existing deep neural networks systems (Feng et al., 2018; Ghorbani et al., 2019; Oana-Maria et al., 2019b).”
Data Shapley: Equitable Valuation of Data for Machine Learning (2019, background): “These methods, however, do not satisfy any equitability conditions, and also have been shown to have robustness issues (Ghorbani et al., 2017).”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “In practice, however, the stability of both gradient-based [8, 54, 66, 122] and counterfactual-based [104, 106] explanation systems has been called into question.”
Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation (2019, background): “In fact, some recent works showed that attribution methods can actually produce un-reliable or misleading results, despite these being visually appealing to humans (Kindermans et al., 2017; Ghorbani et al., 2019; Adebayo et al., 2018; Nie et al., 2018).”
Towards Explanation of DNN-based Prediction with Guided Feature Inversion (2018, background): “[14] demonstrated that several recent interpretation models, including [20, 35, 36, 40], are fragile under adversarial attack.”
Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2021, background): “For instance, Ghorbani et al. [169] demonstrated that a small change in the input sample caused large ﬂuctuations in the output representations generated by XML.”