Explanations can be manipulated and geometry is to blame

By Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, K. Müller, Pan Kessel. 2019

401 citations recorded by the source corpus.

Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

Subfields: Artificial Intelligence, Health Informatics, Information Systems and Management, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Information Systems and Management

In Safety, Robustness & Interpretability

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “Furthermore, explanation can even be manipulated arbitrarily [166].”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, background): “Therefore, one should strive to better understand the effects of wrong feedback and even adversarial attacks [33] on XIL.”
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations (2019, background): “For instance, there has been work demonstrating that explanations can be unstable, changing drastically even with small perturbations to inputs (Dombrowski et al. 2019; Ghorbani, Abid, and Zou 2019).”
Explainable machine learning in deployment (2019, background): “tion between adversarial robustness and model interpretations [18, 21, 23, 59, 69].”
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods (2019, background): “[5] show that by modifying inputs in a way that is imperceptible to humans, they can arbitrarily change saliency maps.”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “In practice, however, the stability of both gradient-based [8, 54, 66, 122] and counterfactual-based [104, 106] explanation systems has been called into question.”
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications (2021, background): “Relatedly, even when the model remains unchanged, inputs could be perturbed imperceptibly to produce arbitrary explanations [39].”
Obtaining genetics insights from deep learning via explainable artificial intelligence (2022, neutral): “Notably, integrated Hessians cannot be applied to neural networks that use an activation function without an informative second derivative, such as the widely used ReLU activation function 75 .”
Higher-Order Explanations of Graph Neural Networks via Relevant Walks (2020, background): “mark—is not developed with built-in robustness to adversarial attacks [62].”
Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond (2020, background): “Re-latedly, even when the model remains unchanged, inputs could be perturbed imperceptibly to produce arbitrary explanations [32].”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “Beyond doubt, understanding the brittleness of deep networks (also in the context of their explanations [515]) will be critical for their adoption in AD applications that involve malicious attackers, such as fraudsters or network intruders.”
Debugging Tests for Model Explanations (2020, methodology): “, 2017); and finally, 3) We select methods that have been shown to have desirable against manipulation (SmoothGrad) (Dombrowski et al., 2019; Yeh et al., 2019).”
How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods (2019, background): “Dombrowski et al. (2019) show that by modifying inputs in a way that is imperceptible to humans, they can arbitrarily change saliency maps.”
Fairwashing Explanations with Off-Manifold Detergent (2020, background): “In (Dombrowski et al., 2019) and (Ghorbani et al., 2019), it is shown that explanations can be manipulated by an infinitesimal change in input while the output of the network is approximately unchanged.”
You Shouldn't Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods (2020, background): “This was analysed in terms of the geometry of the learned function [9].”