Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning

By Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, R. Caruana, Hanna M. Wallach, Jennifer Wortman Vaughan. 2020

609 citations recorded by the source corpus.

Machine learning (ML) models are now routinely deployed in domains ranging from criminal justice to healthcare. With this newfound ubiquity, ML has moved beyond academia and grown into an engineering discipline. To that end, interpretability tools have been designed to help data scientists and machine learning practitioners better understand how ML models work. However, there has been little evaluation of the extent to which these tools achieve this goal. We study data scientists' use of two existing interpretability tools, the InterpretML implementation of GAMs and the SHAP Python package. We conduct a contextual inquiry (N=11) and a survey (N=197) of data scientists to observe how they use interpretability tools to uncover common issues that arise when building and evaluating ML models. Our results indicate that data scientists over-trust and misuse interpretability tools. Furthermore, few of our participants were able to accurately describe the visualizations output by these tools. We highlight qualitative themes for data scientists' mental models of interpretability tools. We conclude with implications for researchers and tool designers, and contextualize our findings in the social science literature.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “Kaur et al. (2020); Poursabzi-Sangdeh et al. (2018) (among others) have performed extensive human experiments on interpretable models and posthoc explanations of black box models, with some interesting and sometimes non-intuitive results.”
Manipulating and Measuring Model Interpretability (2018, methodology): “[49] studied data scientists’ use of two specific interpretability tools (the InterpretML [81] implementation of generalized additive models and the SHAP Python package), finding that data scientists over-trust and misuse these tools.”
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020, background): “As noted by Wiegreffe and Pinter (2019); Vashishth et al. (2019), although in the context of attention solely.”
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks (2021, background): “Additionally, it has been shown that numerical or visual explanatory methods, in some cases, can be confusing even for data scientists [24] and can pose problems even for explaining trivial models [13].”
Explainable Deep Learning Models in Medical Image Analysis (2020, background): “A recent study compared the understanding of explanations amongst data scientists [71].”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, methodology): “Similar interactive tools have been used to support data scientists in debugging machine learning models and improving model performance [24, 26, 69].”
Rationale-Inspired Natural Language Explanations with Commonsense (2021, background): “extractive rationales may not paint the full picture or may be hard to comprehend even by experts (Kaur et al., 2020; Camburu et al., 2021; Alufaisan et al., 2021; Wiegreffe et al., 2020).”
Interpretable Machine Learning (2019, background): “105,106 Wider ML accountability tools In addition to technical approaches to interpretable ML, many stakeholders have called for wider accountability mechanisms to ensure that ML systems are designed and deployed in an ethical and responsible way.”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “As a result, various studies have evaluated the effect of explanations from different dimensions, including whether the explanation improves users’ trust on the AI [34, 68] or enables users to simulate the model predictions [13, 55], or assisting developers to debug and train a better ML model [7, 3”
On the Privacy Risks of Model Explanations (2019, background): “The layer sizes are [20, 20, 20, 20, 2].”
How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India (2022, background): “Prior work calls for aligning transparency with user needs [42, 75].”
I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI (2021, background): “The provided XAI explanations encouraged the users to apply their heuristic thinking instead of activating their analytical thinking [25].”
Explanation-Based Human Debugging of NLP Models: A Survey (2021, background): “Moreover, some users may over trust the explanations (without fully understanding them) only because the tools generating them are publicly available, widely used, and showing appealing visualizations (Kaur et al., 2020).”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Recent studies also reported detrimental effects of explanations for AI system users including inducing over-trust or over-estimation of model capabilities [50, 90, 95], and increasing cognitive”
Epistemic values in feature importance methods: Lessons from feminist epistemology (2021, background): “heartening that user studies on ML explanations have started appearing in major HCI conferences [36, 41].”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, background): “Many studies have found that cognitive biases (particularly anchoring bias and automation bias more broadly) can lead users to misunderstand model behavior and place unwarranted trust in AI systems [20, 34, 52, 55].”