Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
By Cynthia Rudin. 2018
9,528 citations recorded by the source corpus.
Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.
Subfields: Artificial Intelligence, Health Informatics, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Noah A. Smith
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Quoc V. Le
In Health Informatics
- Gesina Schwalbe
- Bettina Finzel
- Octavio Loyola-González
- Timo Speith
- S. Lo Piano
- Adrian Weller
- David Watson
- Luciano Floridi
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Cynthia Rudin
- Wojciech Samek
- K. Müller
- Q. Liao
- Sarah Wiegreffe
- Grgoire Montavon
Citations in the corpus
- Diagnosing Gender Bias in Image Recognition Systems (2020, background): “A system is a black box either if its technical design is sufficiently complex that human users cannot interpret the meaning of the inner workings or if the details of the system’s design and construction are hidden from users, for example, as corporate trade secrets (Rudin 2019).”
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “It is in the confluence of these three design drivers where the guidelines postulated in [383] (and other studies in this same line of thinking [384] ) recommend first the consideration of standard interpretable models rather than sophisticated yet opaque modeling methods.”
- Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “In particular, there are many serious problems with the use of explaining black boxes posthoc, as outlined in several papers that have shown why explaining black boxes can be misleading and why explanations do not generally serve their intended purpose (Rudin, 2019; Laugel et al., 2019; Lakkaraju an”
- Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, methodology): “a white-box approach) from a practical point of view [3], [4].”
- Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning (2021, background): “However, some concerns have been brought up (Rudin, 2019; Laugel et al., 2019) on potential issues of black-box explainers.”
- Ethical Implementation of Artificial Intelligence to Select Embryos in In Vitro Fertilization (2021, background): “One key point that has emerged from past research is that as long as one can design the interpretability metric carefully to match the domain, interpretable models tend not to lose accuracy relative to their black-box counterparts [26, 27].”
- A case-based interpretable deep learning model for classification of mass lesions in digital mammography (2021, background): “Posthoc explanations can be problematic; for instance saliency maps highlight regions of the image, but can be unreliable and misleading, as they tend to highlight edges and do not show what computation is actually done with the highlighted pixels [40, 2, 3].”
- Mathematical optimization in classification and regression trees (2021, background): “Instead, and as advocated by Rudin (2019), it is better to work with models that can derive local explanations directly, as is the case for Neural Networks (Gevrey et al.”
- Characterizing Fairness Over the Set of Good Models Under Selective Labels (2021, methodology): “When defined in this manner, the set of good models is also called the “Rashomon set” (Rudin, 2019; Fisher et al., 2019; Dong & Rudin, 2020; Semenova et al., 2020).”
- Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “Such a hierarchical, logical model aids interpretability [14, 43], since a tree has various conceptual advantages compared to a linear bag-of-prototypes: a tree enforces a sequence of steps and it supports negative associations (i.”
- Application of Artificial Intelligence Technology in Oncology: Towards the Establishment of Precision Medicine (2020, background): “In Europe, the General Data Protection Regulation (GDPR), which came into force in May 2018, included an article (Article 22) requiring the transparency of AI; consequently, it is necessary to address the black box issue in terms of GDPR regulation compliance [162].”
- This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition (2020, methodology): “However, the black box nature of neural networks prevents a human to assess the model’s decision making process, which is especially problematic in domains with high stakes decisions [36].”
- Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “Some authors have suggested that this approach is the only acceptable solution for high-stakes decision making due to both technical and conceptual limitations in trying to create explanations for uninterpretable models.(39) Indeed, much current research into producing ‘‘post hoc’’ explanations(40) ”