The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations

By Thibault Laugel, Marie-Jeanne Lesot, C. Marsala, Xavier Renard, Marcin Detyniecki. 2019

231 citations recorded by the source corpus.

Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “In particular, there are many serious problems with the use of explaining black boxes posthoc, as outlined in several papers that have shown why explaining black boxes can be misleading and why explanations do not generally serve their intended purpose (Rudin, 2019; Laugel et al., 2019; Lakkaraju an”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, neutral): “In contrast to post-hoc explanations, which approximate a trained model or its output [37, 31], a ProtoTree is inherently interpretable since it directly incorporates interpretability in the structure of the predictive model [37].”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “In practice, however, the stability of both gradient-based [8, 54, 66, 122] and counterfactual-based [104, 106] explanation systems has been called into question.”
Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations (2021, background): “However, post-hoc explanatory methods have been shown to have certain downsides (Adebayo et al., 2018; Slack et al., 2020; Laugel et al., 2019; Camburu et al., 2021; Wiegreffe et al., 2021; Camburu et al., 2019).”
Counterfactual Explanations for Machine Learning: A Review (2020, background): “[75] says that if the explanation is not based on training data, but the artifacts of non-robustness of the classifier, it is unjustified.”
Model-Agnostic Linear Competitors - When Interpretable Models Compete and Collaborate with Black-Box Models (2019, background): “However serious concerns have been brought up (Rudin 2019, A¨ıvodji et al. 2019, Thibault et al. 2019) on potential issues of black-box explainers since explainers only approximate but do not characterize exactly the decision-making process of a black-box model, often yielding an imperfect ﬁdelity…”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “We have a better understanding of weaknesses of IML methods in general [75,79], but also specifically for methods such as permutation feature importance [51,110,7,111], Shapley values [57,113], counterfactual explanations [63], partial dependence plots [51,50,7] and saliency maps [2].”
Explainability fact sheets: a framework for systematic assessment of explainable approaches (2019, background): “Since we have not seen many data points in that region, we should not trust the explanation without further investigation [30, 40].”
Drug discovery with explainable artificial intelligence (2020, background): “130 Such approaches (herein referred to as ‘ self-explaining ’) would promote veriﬁcation and error analysis, and be directly linkable with domain knowledge.”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “A general evaluation metric for post-hoc methods for explainability was presented in [323] based on the risk of generating unjustified ‘counterfactual examples’ which are instances that do not depend on previous knowledge but are artefacts of the classifier.”
A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence (2021, methodology): “raise the issue of justification for counterfactual explanation [144].”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “Laugel et al. [211] say that if the explanation is not based on training data, but the artifacts of non-robustness of the classifier, it is unjustified.”