Evaluating Explanation Without Ground Truth in Interpretable Machine Learning

By Fan Yang, Mengnan Du, Xia Hu. 2019

81 citations recorded by the source corpus.

Interpretable Machine Learning (IML) has become increasingly important in many real-world applications, such as autonomous cars and medical diagnosis, where explanations are significantly preferred to help people better understand how machine learning systems work and further enhance their trust towards systems. However, due to the diversified scenarios and subjective nature of explanations, we rarely have the ground truth for benchmark evaluation in IML on the quality of generated explanations. Having a sense of explanation quality not only matters for assessing system boundaries, but also helps to realize the true benefits to human users in practical settings. To benchmark the evaluation in IML, in this article, we rigorously define the problem of evaluating explanations, and systematically review the existing efforts from state-of-the-arts. Specifically, we summarize three general aspects of explanation (i.e., generalizability, fidelity and persuasibility) with formal definitions, and respectively review the representative methodologies for each of them under different tasks. Further, a unified evaluation framework is designed according to the hierarchical needs from developers and end-users, which could be easily adopted for different scenarios in practice. In the end, open problems are discussed, and several limitations of current evaluation techniques are raised for future explorations.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “Explanation plausibility may be described as a subjective satisfaction with how a given explanation justiﬁes a label/answer (Yang et al., 2019).”
Robust Feature-Level Adversaries are Interpretability Tools (2021, background): “One challenge in the explainable AI literature is to develop interpretations that go beyond seemingplausible and stand up to scrutiny [66].”
Learning Credible Deep Neural Networks with Rationale Regularization (2019, background): “Most current work focus on augmenting DNNs with interpretability [8], [19], [20], while employing explanation to enhance the performance of DNN models has seldom been explored.”
Evaluating and Aggregating Feature-based Model Explanations (2020, background): “We do not delve into human-centered evaluation of explanation functions either; see [Gilpin et al. , 2018; Poursabzi-Sangdeh et al. , 2018; Yang et al. , 2019] for detailed discussions.”
On Completeness-aware Concept-Based Explanations in Deep Neural Networks (2019, background): “The notion of sufﬁciency is also referred to as “completeness” of explanations, as in [9, 10].”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “, 2019), evaluation (Hoffman et al., 2018; Yang et al., 2019), or a combination of these topics (Lipton, 2018; Doshi-Velez and Kim, 2017; Adadi and Berrada, 2018; Murdoch et al.”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “It can reflect two facets of certainty: i) a confidence measure of the black box prediction [17, 32, 214, 289], or ii) the truthfulness or likelihood of the explanation [17, 166, 233].”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “Highlights are usually evaluated following two criteria: (i) plausibility, according to humans, how well a highlight supports a predicted label [133, 29], and (ii) faithfulness or fidelity, how accurately a highlight represents the model’s decision process [6, 127].”
On Concept-Based Explanations in Deep Neural Networks (2019, background): “The notion of explanations that are sufficient for prediction is also called the “completeness” of explanations [11], which is acknowledged to be valuable for evaluating explanations [29].”
Fusing Multi‐ and Hyperspectral Satellite Data for Harmful Algal Bloom Monitoring With Self‐Supervised and Hierarchical Deep Learning (2025, unknown): “This study provides a baseline from which we can continue to expand validation and application to larger spatiotemporal regions to better understand strengths, limitations, and related factors.”
Development and Application of Self-Supervised Machine Learning for Smoke Plume and Active Fire Identification from the FIREX-AQ Datasets (2025, unknown): “This is a fairly common problem within the remote sensing domain and one we aim to help solve with the collective incorporation of self-supervised learning, subject matter expert domain knowledge, and large amounts of data [48,49].”