Machine Learning Interpretability: A Survey on Methods and Metrics

By D. V. Carvalho, Eduardo Marques Pereira, Jaime S. Cardoso. 2019

1,674 citations recorded by the source corpus.

Machine learning systems are becoming increasingly ubiquitous. These systems’s adoption has been expanding, accelerating the shift towards a more algorithmic society, meaning that algorithmically informed decisions have greater potential for significant social impact. However, most of these accurate decision support systems remain complex black boxes, meaning their internal logic and inner workings are hidden to the user and even experts cannot fully understand the rationale behind their predictions. Moreover, new regulations and highly regulated domains have made the audit and verifiability of decisions mandatory, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is indispensable. The research community has recognized this interpretability problem and focused on developing both interpretable models and explanation methods over the past few years. However, the emergence of these methods shows there is no consensus on how to assess the explanation quality. Which are the most suitable metrics to assess the quality of an explanation? The aim of this article is to provide a review of the current state of the research field on machine learning interpretability while focusing on the societal impact and on the developed methods and metrics. Furthermore, a complete literature review is presented in order to identify future directions of work on this field.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ViCE: visual counterfactual explanations for machine learning models (2020, neutral): “Previous works [1–4, 6, 10, 15, 17] provide an overview of methods, opportunities and challenges in this area.”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “With the rise of opaque and poorly-understood machine learning models in the field of AI, trust is often cited as a key desirable property of the interaction between any user and the AI [8, 11, 71, 76].”
A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, background): “Interpretabil-ity (Carvalho et al., 2019) refers to the ability of a user to understand the ensemble outcomes.”
Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond (2021, methodology): “Researchers have leveraged the feature importance to explain the characteristics and significance of the extracted features and the correlations among features and between features and the outcomes for providing interpretability for AI models [2,141,142].”
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques (2019, methodology): “Besides these three criteria, Carvalho et al. [15] add a criterion on the stage of model development where an explainability method is applied, i.e., whether it is before ( pre-model ), during ( in-model ), or after ( post-model ) building the machine learning model.”
The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review (2024, background): “According to Carvalho et al. (2019), such transparency is pivotal for enhancing understanding of AI dialogue systems, laying a solid foundation for the challenges and advantages of algorithmic decisions, and ensuring that decision-making processes are both accountable and fair.”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences (2020, methodology): “he development of intelligent systems that optimize the choice of explanation, such as interactive or personalized explanations [4,81,94]. Toward this goal, many sought to deﬁne the desiderata of XAI [19,40,45,77], including ﬁ- delity, completeness, robustness, etc. Despite the conceptual discussion”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, methodology): “1Regarding terminology: ‘interpretability’ and ‘explainability’ are closely related and often used interchangeably in the XAI context [29, 32].”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “For interested readers, we suggest several recent papers that provided deeper technical surveys [6, 11, 20, 44].”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “For high-level introductions to the field we refer to [16-18].”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, neutral): “Research-wise, the field is maturing in terms of methods surveys [75,41,120,96,1,6,23,15], further consolidation of terms and knowledge [42,22,82,97,88,17], and work about defining interpretability or evaluation of IML methods [74,73,95,49].”
Benchmarking and survey of explanation methods for black box models (2021, background): “Further taxonomies and definitions are presented in (Murdoch et al. 2019; Carvalho et al. 2019).”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “Researchers [29, 47, 81, 101, 107, 183] also proposed many other desiderata for interpretations, interpretation algorithms or interpretability, such as fairness, privacy, reliability, robustness, causality, trust, fidelity, faithfulness, transferability, informativeness, transparency, plausibility, ”
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research (2022, background): “CATEGORIZATIONS OF XAI According to [53], [54], the XAI categories can be structured in a variety of aspects shown in Figure 6.”