On the Robustness of Interpretability Methods

By David Alvarez-Melis, Tommi S. Jaakkola. 2018

670 citations recorded by the source corpus.

We argue that robustness of explanations---i.e., that similar inputs should give rise to similar explanations---is a key desideratum for interpretability. We introduce metrics to quantify robustness and demonstrate that current methods do not perform well according to these metrics. Finally, we propose ways that robustness can be enforced on existing interpretability approaches.

Subfields: Artificial Intelligence, Control and Systems Engineering, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Explaining Predictions from Tree-based Boosting Ensembles (2019, background): “Indeed, since the local model is an approximation, there is no guarantee that it is appropriately representative of the original model, especially in other parts of the input space [1].”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, positive): “In contrast to local explanations, which explain a single prediction and can be unstable and contradicting [3, 27], global explanations enable simulatability [35].”
Do Not Trust Additive Explanations (2019, background): “That is why it is important to assess not only the accuracy of a model but also assess the accuracy of such explanations (Alvarez-Melis and Jaakkola, 2018).”
TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2018, background): “Most interpretability methods suffer from this inherent weakness due to reliance on first-order gradients [31].”
Towards Robust Interpretability with Self-Explaining Neural Networks (2018, background): “However, point-wise interpretations generally do not compare explanations obtained for nearby inputs, leading to unstable and often contradicting explanations [1].”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “, 2018) or the “robustness” (Alvarez-Melis and Jaakkola, 2018) of the interpretation method.”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “stability of both gradient-based [8, 54, 66, 122] and counterfactualbased [104, 106] explanation systems has been called into question.”
Model-Agnostic Linear Competitors - When Interpretable Models Compete and Collaborate with Black-Box Models (2019, methodology): “ents for dierent classes (see Figure 4) so that users can easily understand what features dierentiate one class from the others. Figure 5 Means and Standard Deviations of Coecients of LIME Second, Alvarez-Melis and Jaakkola (2018) showed that LIME’s explanation of two close (similar) instances can v”
Analyzing the Interpretability Robustness of Self-Explaining Models (2019, background): “However, Ghorbani et al. (2019) and Alvarez-Melis & Jaakkola (2018) recently showed that post hoc methods are fragile, where small changes in the input cause signiﬁcant changes in the interpretations without changing the model output.”
Acquisition of chess knowledge in AlphaZero (2021, methodology): “While many of these methods are shown to be useful for end-tasks, more recent studies called some of these methods into question in terms of their validity [33, 34], robustness [35] or their formulation [36].”
"Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations (2019, background): “L G ] 4 J un 2019 been pointed out by Alvarez-Melis & Jaakkola, who showed that while LIME explanations can be stable when explaining linear models, for nonlinear models this is not always the case (Alvarez-Melis & Jaakkola, 2018).”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Intuitively, if the input being explained is modified slightly-subtly enough to not change the prediction of the model then the explanation must not change much either [91].”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Moreover, we should note that the adversarial attacks do not only aim at trained models [30], but the interpretations [8, 56, 71].”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, background): “Given a large-scale latent diffusion model for text-to-image synthesis, which parts of a generated image does each word influence most? Attribution approaches in computer vision are primarily perturbation- and gradient-based [10, 11], where saliency maps are either constructed from the first derivat”
Explainable Deep Learning: A Field Guide for the Uninitiated (2020, background): “The first is to devise an objective metric or benchmark to evaluate the explanations without human intervention (Samek et al., 2016; Hooker et al., 2019; Vu et al., 2019; Adebayo et al., 2018; Alvarez-Melis & Jaakkola, 2018).”
RL-LIM: Reinforcement Learning-based Locally Interpretable Modeling (2019, background): “…for LIME is the need for a meaningful distance metric to determine neighborhoods, as simple metrics like Euclidean distance may yield poor ﬁdelity in some cases and the estimation can be highly-sensitive to normalization (Alvarez-Melis & Jaakkola, 2018) especially with categorical variables.”