Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models

By Fred Hohman, Andrew Head, R. Caruana, R. Deline, Steven Mark Drucker. 2019

281 citations recorded by the source corpus.

Without good models and the right tools to interpret them, data scientists risk making decisions based on hidden biases, spurious correlations, and false generalizations. This has led to a rallying cry for model interpretability. Yet the concept of interpretability remains nebulous, such that researchers and tool designers lack actionable guidelines for how to incorporate interpretability into models and accompanying tools. Through an iterative design process with expert machine learning researchers and practitioners, we designed a visual analytics system, Gamut, to explore how interactive interfaces could better support model interpretation. Using Gamut as a probe, we investigated why and how professional data scientists interpret models, and how interface affordances can support data scientists in answering questions about model interpretability. Our investigation showed that interpretability is not a monolithic concept: data scientists have different reasons to interpret models and tailor explanations for specific audiences, often balancing competing concerns of simplicity and completeness. Participants also asked to use Gamut in their work, highlighting its potential to help data scientists understand their own data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ecological Modeling

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, studying the efficacy of interpretability tools or exploring possible interfaces–would be welcome, as would further exploration of how practitioners currently use such tools [81] [82] [83] [78] [84].”
Manipulating and Measuring Model Interpretability (2018, background): “This problem is exacerbated by the fact that machine learning models have many different types of stakeholders and these stakeholders may have different needs in different scenarios [44, 99, 100].”
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations (2020, methodology): “Hence, GAMs are often considered to be intelligible [9, 26, 39] for a wide variety of applications.”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, background): “For model development or debugging, research suggests that users often need a range of explanations for different levels of the model behaviors to perform a comprehensive diagnosis [51, 52, 77].”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, methodology): “By design, however, GAMut aims only at the analysis of a particular type of machine learning algorithm (generalized additive models).”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, background): “, data scientists [22]) and the consequences of imperfect interpretation (e.”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, methodology): “Similar interactive tools have been used to support data scientists in debugging machine learning models and improving model performance [24, 26, 69].”
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques (2019, background): “There is comparatively much less work in the other two categories, notwithstanding calls for interactive explanation [32, 42, 61], and we hope that the taxonomy inspires more research in those directions.”
Bridging the Gap Between Ethics and Practice (2020, background): “For complex decisions, more elaborate user interfaces and data analytics should clarify which of the many features are the most relevant [49], even in deep learning for image understanding [50].”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, background): “[141] developed an interactive userinterface for providing explanations to data scientists.”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “Typical methods include ask people to rank the input data features based on their inuence to overall predictions [18, 33], to indicate the direction of change in the model’s prediction when a feature’s value is altered [16, 18, 29], to simulate the model’s predictions [16, 24, 40, 47, 59], to answer”
The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2020, background): “Furthermore, outlining how different explanations can be best combined in a user interface [80], and how these combined AI systems should be evaluated are open research problems.”