Towards A Rigorous Science of Interpretable Machine Learning
By Finale Doshi-Velez, Been Kim. 2017
5,317 citations recorded by the source corpus.
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Noah A. Smith
- Jianfeng Gao
- Yejin Choi
- Quoc V. Le
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Wojciech Samek
- Been Kim
- K. Müller
- Yonatan Belinkov
- Ana Marasovic
- Sarah Wiegreffe
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Evaluating the interpretability of a model itself as an additional dimension is extrinsic and task-specific (Doshi-Velez and Kim, 2017) and thus out of scope for this work.”
- ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “evaluation of our explanations using forward simulation (Doshi-Velez and Kim, 2017), where we ask humans to predict the system output based on the explanations.”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “…attend to when they make decisions (Greydanus et al., 2018), evaluating models’ sensitivity to the presence/intensity of high-level human concepts (Kim et al., 2017), optimizing a model to be more interpretable with humans in the loop (Lage et al., 2018), translating neural activations into…”
- Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “ our primary foci for safety in dialogue: (1) providing learning performance guarantees; (2) proper objective specification; (3) model interpretability (Amodei et al. 2016; Garcıa and Fern´andez 2015; Doshi-Velez and Kim 2017). Furthermore, to understand why these aspects of AI safety are relevant to”
- When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2021, background): “Improving models in this manner is a natural goal of approaches using explanations, since one purpose of an explanation is to communicate a mental model (Doshi-Velez & Kim, 2017; Miller, 2019).”
- Measuring Association Between Labels and Free-Text Rationales (2020, background): “Human simulatability (Doshi-Velez and Kim, 2017) has a rich history in machine learning inter-”
- WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “Instead, we refer the interested reader to the surveys provided by (Doshi-Velez & Kim, 2017; Molnar, 2019; Guidotti et al., 2018).”
- CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “ually create examples. 3.2 Model Interpretations and Debiasing in NLP Model interpretability is the degree to which a human can consistently predict the model’s outcome (Kim, Khanna, and Koyejo 2016; Doshi-Velez and Kim 2017; Lipton 2018). The more easily interpretable a machine learning model is, t”
- Analysis Methods in Neural Language Processing: A Survey (2018, background): “With the increasing public demand for explaining algorithmic choices in machine learning systems (Doshi-Velez and Kim, 2017; Doshi-Velez et al., 2017), there is pressing need for progress in this direction.”
- Attention is not not Explanation (2019, methodology): “Doshi-Velez and Kim (2017) break down interpretability into further subcategories, depending on the amount of human involvement and the difficulty of the task.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “A previous survey has attempted to define taxonomies and best practices for a “strong science” of interpretability [74].”
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, methodology): “We hope the work here will help push the field towards a more rigorous way to use interpretability methods, a point which will become increasingly important as interpretable machine learning develops as a field (Doshi-Velez & Kim, 2017; Murdoch et al., 2019).”
- A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “In essence, [9] suggests that we create a matrix whose rows are different real-world tasks (e.g. pneumonia detection), columns are different methods (e.g. decision tree with different depths) and the entries are the performance of the methods on some end-task .”
- Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “…2012, 2011, 2013, 2015; Nataksu, 2004; Ribiero, et al., 2016a,b; Sheh and Monteath, 2008; Treu, 1998; Wickens, 1994), and the challenges for the evaluation of explanation systems (Doshi-Velez and Kim, 2017ab; Gilpin et al., 2018; Feldmann, 2018; Miller, 2017; Miller, Howe, and Sonenberg, 2017).”
- Fairwashing: the risk of rationalization (2019, background): “In the context of machine learning, interpretability can be defined as the ability to explain or to provide the meaning in understandable terms to a human [11].”
- On the Art and Science of Machine Learning Explanations (2018, background): “Others have defined key terms and put forward general motivations for better interpretability of machine learning models (and advocated for stronger scientific rigor in certain cases) [9], [11], [13], [18].”