Please Stop Explaining Black Box Models for High Stakes Decisions

By Cynthia Rudin. 2018

236 citations recorded by the source corpus.

Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward - it is to design models that are inherently interpretable.

Subfields: Artificial Intelligence, Health Informatics, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, methodology): “trade-off between accuracy and explainability [3], [4], [106].”
Fairwashing: the risk of rationalization (2019, background): “While many recent works have focused on black-box explanations, their use can be problematic in high stakes decision-making systems (Reddix-Smalls, 2011; Wexler, 2017b) as explanations can be unreliable and misleading (Rudin, 2018; Melis & Jaakkola, 2018).”
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2018, background): “A preliminary version of this manuscript appeared at a workshop, entitled “Please Stop Explaining Black Box Machine Learning Models for High Stakes Decisions” [13].”
In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes (2019, background): “, [3, 34]), we consider interpretability to be a more desirable characteristic of a model given recent criticisms of black box models [36].”
Interpretable and Steerable Sequence Learning via Prototypes (2019, background): “Therefore it is often desirable to have models with inherent interpretability in many application scenarios [29].”
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, methodology): “However, there seems to have been few works [Lipton, 2017; Rudin, 2018] questioning the post-hoc paradigm itself, and wondering about the risks of generating explanations that are disconnected from the ground truth.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, neutral): “Rudin (2018) ar-”
Model-Agnostic Counterfactual Explanations for Consequential Decisions (2019, background): “To make things worse, while the verdict is still out as to what constitutes a good explanations [7, 10, 16, 19, 23, 24], there already exists clearly deﬁned legal requirements for explanations in the context of consequential decision making.”
Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “There has been an increasing trend in healthcare, criminal justice, and other regulated domains to leverage ML for high-stakes prediction applications, which deeply impact human lives [9].”
The Scientific Method in the Science of Machine Learning (2019, background): “k & Cisse, 2018; Zhang et al., 2017). These inconsistencies in performance are particularly troubling in deployed systems, whose errors have real world implications (Buolamwini & Gebru, 2018; Rudin, 2018; Zech et al., 2018). Many have identiﬁed various social factors within the research ecos”
A Survey on the Explainability of Supervised Machine Learning (2020, background): “Rudin (2018) claims that this is often not true especially when considering structured data with meaningful features.”
Challenges and frontiers in abusive content detection (2019, background): “Rudin argues that a better approach is to ‘design models are inherently interpretable’ (Rudin, 2018, p. 1).”
SIRUS: making random forests interpretable (2019, background): “Nonetheless, even if they are useful, such treatments only provide partial information and can be diﬃcult to operationalize for critical decisions (Rudin, 2018).”
Recalibrating classifiers for interpretable abusive content detection (2020, background): “It may even require complete rethinking of how classiﬁers are developed, given the difﬁculties of post-hoc rationalisation and the potential for poor explanations to confuse end users (Rudin, 2018).”
Bayesian Workflow (2020, methodology): “…understand, hence the need for exploratory model analysis (Unwin et al., 2003, Wickham, 2006) and explainable AI (Chen et al., 2018, Gunning, 2017, Rudin, 2018), which complements methods for evaluating, comparing, and averaging models using a range of interrelated approaches, including cross…”