A Survey of Methods for Explaining Black Box Models
By Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, Fosca Giannotti. 2018
5,098 citations recorded by the source corpus.
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Noah A. Smith
- Yoshua Bengio
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Yoav Goldberg
In Safety, Robustness & Interpretability
- Longo Luca
- Giulia Vilone
- Wojciech Samek
- K. Müller
- Xisen Jin
- Grgoire Montavon
- Ana Marasovic
- Sarah Wiegreffe
Citations in the corpus
- Refining Language Models with Compositional Explanations (2021, methodology): “1) produced by a post-hoc model explanation algorithm [13].”
- WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “Instead, we refer the interested reader to the surveys provided by (Doshi-Velez & Kim, 2017; Molnar, 2019; Guidotti et al., 2018).”
- Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Although some recent surveys [8,10,13–17] summarize the upsurge of activity in XAI across sectors and disciplines, this overview aims to cover the creation of a complete unified framework of categories and concepts that allow for scrutiny and understanding of the field of XAI methods.”
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, methodology): “A recent survey on methods for explaining black-box models [76] outlined a taxonomy to provide classifications of the main problems with opaque algorithms.”
- Explaining Predictions from Tree-based Boosting Ensembles (2019, background): “Recent work on local explanations is usually conducted in either a modelagnostic or model-specic way [5].”
- CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models (2019, methodology): “Other general methods on explainability, fairness and robustness have been described by [Guidotti et al., 2018b],[Binns, 2017], and [Akhtar and Mian, 2018] respectively.”
- Fairwashing: the risk of rationalization (2019, methodology): “In this paper, we focus on two black-box explanation tasks, namely model explanation and outcome explanation (Guidotti et al., 2018).”
- On the Art and Science of Machine Learning Explanations (2018, background): “Others have defined key terms and put forward general motivations for better interpretability of machine learning models (and advocated for stronger scientific rigor in certain cases) [9], [11], [13], [18].”
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “[63] reviewed methods for explaining black-box models at a large scale including data mining as well as machine learning.”
- Local Rule-Based Explanations of Black Box Decision Systems (2018, background): “Recently, the research of methods for explaining black box decision systems has caught much attention [17].”
- Interpreting tree ensembles with inTrees (2014, neutral): “[19] provided a thorough review on that.”
- Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, background): “fraction of test images for which the soft and hard strategy make the same classification [19]) is practically 1.”
- This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition (2020, methodology): “Following this demand on understanding automated decision making, explainable Artificial Intelligence (XAI) has been actively researched [1, 3, 16].”
- Computing Optimal Decision Sets with SAT (2020, background): “Hence the rising interest in explainable artificial intelligence (XAI) [4, 14, 16, 18, 22, 23, 26, 29, 30, 36–38, 42, 43, 48, 49, 52], including research programs [3, 24] and legislation [17, 21].”
- ViCE: visual counterfactual explanations for machine learning models (2020, neutral): “Previous works [1–4, 6, 10, 15, 17] provide an overview of methods, opportunities and challenges in this area.”
- Learning Fair Rule Lists (2019, background): “Two main approaches have emerged in the literature to facilitate the understanding of machine learning models: black-box explanation and transparent-box design [27, 42, 44, 48].”
- The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations (2019, methodology): “[ Guidotti et al. , 2018a ] ) look for the L 2 (resp.”