Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

By Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju. 2019

1,143 citations recorded by the source corpus.

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Robustness Analysis Robustness of post-hoc ex-tractive interpretability methods has been studied (Kindermans et al., 2019; Ghorbani et al., 2019; Heo et al., 2019; Zheng et al., 2019; Slack et al., 2020).”
Attributing Fair Decisions with Attention Interventions (2021, background): “In addition, previously it has been shown that interpratability frameworks are vulnerable against adversarial attacks (Slack et al. 2020).”
Predictive Multiplicity in Classification (2019, background): “In this case, reporting evidence of competing models with conflicting predictions would mitigate unwarranted rationalization of the model resulting from fairwashing (Aı̈vodji et al., 2019; Laugel et al., 2019; Slack et al., 2020) or explanation bias (Koehler, 1991).”
Faithful Chain-of-Thought Reasoning (2023, background): “…looks plausible (i.e., convincing to humans) (Jacovi and Goldberg, 2020), this makes it easier for people (e.g., legal practitioners) to over-trust the model (e.g., a recidivism predictor) even if it has implicit biases (e.g., against racial minorities) (Pruthi et al., 2020; Slack et al., 2020).”
Debugging Tests for Model Explanations (2020, background): “Feature attributions can be easily manipulated, providing evidence for a collective ‘weakness’ of current approaches [23, 24, 35, 61].”
Globally-Consistent Rule-Based Summary-Explanations for Machine Learning Models: Application to Credit-Risk Evaluation (2019, positive): “These explanations, however, are not necessarily accurate, nor globally-consistent, and their important features are not necessarily those of the global model; LIME has been heavily criticized for its lack of fidelity in particular (Slack et al. 2020).”
On the Tractability of SHAP Explanations (2020, methodology): “ relies on approximation algorithms to computethe explanations.These approximationalgorithms, however, tend to make simplifying assumptions which can lead to counter-intuitive explanations, see e.g., Slack et al. (2020). We believe that more focus should be given to the computational complexity of S”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, background): “However, post-hoc explanatory methods have been shown to have certain downsides (Adebayo et al., 2018; Slack et al., 2020; Laugel et al., 2019; Camburu et al., 2021; Wiegreffe et al., 2021; Camburu et al., 2019).”
A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME (2023, background): “Third, SHAP is not protected against biased classifier and might generate un-realistic explanations that do not capture the underlying biases [18].”
Explaining by Removing: A Unified Framework for Model Explanation (2020, background): “Our work is among a growing number of papers lending support to this approach (Strobl et al., 2008; Zintgraf et al., 2017; Agarwal and Nguyen, 2019; Aas et al., 2019; Slack et al., 2020; Covert et al., 2020; Frye et al., 2020), but we present a novel perspective on why this choice is justified: besi”
Principles and Practice of Explainable Machine Learning (2020, background): “On the other hand, since these approaches have to be very ﬂexible, a signiﬁcant amount of assumptions and approximations may be made, possibly resulting in poor estimates or undesired side-e ﬀ ects, such as susceptibility to adversarial attacks [74].”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “First, the rapid increase of publications in XAI makes it difficult for researchers to keep up even with research in their discipline, such that they often cannot spare to engage with research of other disciplines (which also has an overwhelming number of publications) [4].”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, background): “For instance, many researchers [263] [264] have proved the fact that it is possible to fool some of the most”