Human Evaluation of Models Built for Interpretability

By Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel J. Gershman, Finale Doshi-Velez. 2019

152 citations recorded by the source corpus.

Recent years have seen a boom in interest in interpretable machine learning systems built on models that can be understood, at least to some degree, by domain experts. However, exactly what kinds of models are truly human-interpretable remains poorly understood. This work advances our understanding of precisely which factors make models interpretable in the context of decision sets, a specific class of logic-based model. We conduct carefully controlled human-subject experiments in two domains across three tasks based on human-simulatability through which we identify specific types of complexity that affect performance more heavily than others–trends that are consistent across tasks and domains. These results can inform the choice of regularizers during optimization to learn more interpretable models, and their consistency suggests that there may exist common design principles for interpretable machine learning systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Manipulating and Measuring Model Interpretability (2018, methodology): “8We emphasize that we are referring to visual information overload that affects attention to items on a display [19], not cognitive load in working memory, which has also been shown to be related to interpretability [1, 60].”
Explaining by Removing: A Unified Framework for Model Explanation (2020, unknown): “Similarly, a different study found that longer explanations required more time and effort to understand, in some cases impeding a user’s ability to draw appropriate conclusions (Lage et al., 2019).”
F1 Is Not Enough! Models and Evaluation towards User-Centered Explainable Question Answering (2020, methodology): “We derive multiple dependent variables from the participants’ ratings, namely completion time (Lim et al., 2009; Lage et al., 2019), several performance variables indicating how well they judged the correctness of the model (fraction of correct ratings, false positive ratio (FP), false negative…”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, methodology): “ user satisfaction and usefulness of explanations in XAI studies. Satisfaction Measures Evaluation Methods User Satisfaction Interview and Self-report ([25, 62, 124, 125]) Likert-scale Questionnaire ([39, 62, 112, 124, 125]) Expert Case Study ([95, 106, 128, 130, 193]) Explanation Usefulness Engagem”
On Completeness-aware Concept-Based Explanations in Deep Neural Networks (2019, background): “Recent work has also focused on evaluations of explanations, ranging from human-centric evaluations [2, 5, 26] to functionally-grounded evaluations [27, 28, 29, 30, 10, 31, 32, 33].”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, background): “is up to them to decide whether and how to use the AI’s assistance to accomplish the task. In contrast, studies that use proxy tasks evaluate how well users are able to simulate the model’s decisions [6, 13, 27, 34] or decision boundaries [29]. In such studies, participants are speci•cally instructe”
Evaluating and Aggregating Feature-based Model Explanations (2020, background): “Though there has been work in qualitatively evaluating feature-based explanation functions on human subjects [Lage et al. , 2019], there has been little exploration into formalizing quantitative techniques for evaluating model explanations.”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, background): “[36] study two aspects of an explanation (length and complexity) via a wizard-of-oz approach in two domains, finding that longer explanations overload people’s cognitive abilities.”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, unknown): “• Studies using proxy tasks to evaluate how well users are able to simulate the model’s decisions [6,70,71] or decision boundaries [72].”
A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning (2018, background): “Multiple papers have reported measurements of users’ understanding of explanations as a proxy for usefulness and interpretability of explanations (Lage et al. 2019; Poursabzi-Sangdeh et al. 2018). Others have measured user-reported trust as a proxy for explanation goodness. For example, Nourani et a”