The mythos of model interpretability

By Zachary Chase Lipton. 2016

4,511 citations recorded by the source corpus.

In machine learning, the concept of interpretability is both important and slippery.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, background): “An explanation is faithful only if it reflects the information that is used for decision making (Lipton, 2018; Jacovi and Goldberg, 2020), which these systems do not guarantee.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “Interestingly, the fact that the present approach generates not only answers but a step-by-step accounting of how they were arrived at means that it provides a limited sort of post hoc interpretability[40] that can be exploited to improve subsequent models.”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, background): “Neural network models are often opaque: they provide limited insight into interpretations of model decisions and are typically treated as “black boxes” (Lipton, 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Explaining specific predictions is recognized as a desideratum in intereptability work (Lipton, 2016), argued to increase the accountability of machine learning systems (Doshi-Velez et al.”
Born Again Neural Networks (2018, background): “We note that in each of these cases, what precisely is meant by interpretability or transparency is often undeclared and the topic remains fraught with ambiguity (Lipton, 2016).”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “While these deep neural networks enable superior performance, their lack of decomposability into intuitive and understandable components makes them hard to interpret [26].”
Attention is not Explanation (2019, background): “We examine the extent to which the (often implicit) narrative that attention provides model transparency (Lipton, 2016).”
Challenges and frontiers in abusive content detection (2019, background): “One challenge here is that ‘explainability’ is itself a contested term and what it entails is not well stipulated (Lipton, 2016).”
Attention is not not Explanation (2019, methodology): “This aligns more closely with the more rigorous (Lipton, 2016, §3.1.1) definition of transparency, or Rudin (2018)’s definition of interpretability: human understanding of the model as a whole rather than of its respective parts.”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “…of the ﬁeld of adversarial machine learning and the emerging ﬁelds of robust artiﬁcial intelligence and inter-pretability of machine learning [101, 102], and we believe that these directions will help our society to get a more conscious understanding of the potential and limits of modern…”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “The opposite of black-box-ness is transparency , i.e., the search for a direct understanding of the mechanism by which a model works [5] .”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Despite intense interest and collective group efforts, we are only seeing limited progress in definitions [8], frameworks [9], visualization [10], and limited experimentation [11].”
The Robust Manifold Defense: Adversarial Training using Generative Models (2017, background): “This relates to interpretability and trust [39, 29, 14] in modern complex models which is an important emerging topic.”
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, background): “We will not attempt to cover all the related works many of which are already presented in the research and survey works we cite [1] [2] [9] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23].”
The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning (2019, methodology): “Indeed, post-hoc explanation has been further sub-divided into the use of (i) textual explanations of system outputs, (ii) visualizations of learned representations and/or models [Erhan et al., 2009; Zeiler and Fergus, 2014] , and (iii) explanations by example [Lipton, 2018].”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “Lipton (2016) Discusses and questions the goal of interpretable models and model explanations.”
Fairwashing: the risk of rationalization (2019, methodology): “Existing methods to achieve explainability include transparent-box design and black-box explanation (also called post-hoc explanation) (Lipton, 2018; Lepri et al., 2017; Montavon et al., 2018; Guidotti et al., 2018).”
On the Art and Science of Machine Learning Explanations (2018, background): “Others have deﬁned key terms and put forward general motivations for better interpretability of machine learning models (and advocated for stronger scientiﬁc rigor in certain cases) [9], [11], [13], [18].”