ModelTracker: Redesigning Performance Analysis Tools for Machine Learning

By Saleema Amershi, David Maxwell Chickering, Steven Mark Drucker, Bongshin Lee, Patrice Y. Simard, Jina Suh. 2015

311 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “In addition, a growing body of research focuses on the design and development of programmer tools that can better support developers in debugging and effectively monitoring the predictive performance of complexML systems [5, 24, 63, 67, 85].”
Software Engineering for Machine Learning: A Case Study (2019, background): “Understanding when and how models fail to make accurate predictions is an active research area [29], [30], [31], which is attracting more attention as ML algorithms and optimization techniques become more complex.”
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models (2018, background): “Typical solutions include visualizing the internal structure or intermediate states of the model to enhance the understanding and interpretation [12,25,41], evaluating and analyzing the performance of models or algorithms [5, 22, 33], and interactively improving the models at different development s”
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models (2016, background): “Some recent interactive machine learning tools [1, 2, 3, 4, 19, 21, 26, 29, 32]) are more algorithm agnostic, but depend on general performance measures like confusion matrices, area under ROC curve (AUC) measures, result distribution of data points, and feature weights according to model independen”
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models (2017, methodology): “, how individual instances contribute to a model’s accuracy) have demonstrated success in a number of machine learning tasks [3,23,29].”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “Amershi et al. (2015) present ModelTracker, a concise visual representation of model performance for use in debugging exercises.”
Understanding Black-box Predictions via Influence Functions (2017, background): “By understanding why a model does what it does, we can hope to improve the model (Amershi et al., 2015), discover new science (Shrikumar et al., 2016), and provide end-users with explanations of actions that impact them (Goodman & Flaxman, 2016).”
Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations (2017, background): “For instance, ModelTracker [1] visualizes predictions and their correctness and how these change when some model parameters are updated.”
Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing (2021, background): “, [4]) could allow DHH users to tolerate errors for less critical sounds (e.”
Visual exploration of machine learning results using data cube analysis (2016, methodology): “Interactive visualizations has been proven to be very effective for finding interesting patterns and spotting anomalies from large, multidimensional data by effectively representing data and allowing users to interact with them [22, 23, 1, 15].”
Towards better analysis of machine learning models: A visual analytics perspective (2017, background): “Researchers have developed visual analytics techniques that diagnose model performance for binary classiﬁers [26], multi-class classiﬁers [2, 27, 28], and topic models [29].”