Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models

By Josua Krause, Adam Perer, Kenney Ng. 2016

418 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Human-Computer Interaction

Citations in the corpus

FairSight: Visual Analytics for Fairness in Decision Making (2019, background): “For example, feature-level auditing methods seek to analyze the feature importance by permutation [11] or quantifying feature interaction [6, 13, 25].”
Model-Agnostic Interpretability of Machine Learning (2016, methodology): “, 2010), perturbing inputs and seeing how the black box model reacts (Strumbelj & Kononenko, 2010; Krause et al., 2016), or both (Ribeiro et al.”
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations (2017, neutral): “…current"labels"such"as"training"the"model,"corpus"genera=on,"etc" Task% Natural%Language% Explana4on% model which describes predictions that were Krause et al., 2016) or by intelligently modifying machine learning algorithm, knowledge the human Figure 2.”
Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks (2016, background): “The recent Prospector system [15] provides a general-purpose tool for practitioners to better understand their ML model and its predictions.”
Meaningful Explanations of Black Box AI Decision Systems (2019, background): “Moreover, appropriate interfaces need to be designed, to convey the answers in meaningful visual and textual forms, as well as visual exploration for advanced users, leveraging also on available visual analytics tools, e.g., (Krause, Perer, and Ng 2016).”
Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda (2018, background): “…as studies on understanding how data scientists work [97], tools to support data scientists in using and understanding machine-learning algorithms [99, 130, 142, 141], visualization techniques for text analysis [34, 35], tools to support interactive data analysis [24, 25, 37] and interactive…”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “Specifically, interactive machine learning tends to involve machine learning practitioners, while our work considers the population that does not have a machine learning background [24, 31, 61, 67].”
Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations (2017, methodology): “A more recent solution is the idea of generating local explanations that are able to explain how a model makes a decision for single instances of a data set [11, 13, 20].”
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models (2018, positive): “[21] use partial dependence diagnostics to explain how features affect the global predictions, while users can interactively tweak feature values and see how the prediction responds to instances of interest.”
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models (2017, methodology): “Researchers have utilized features to visually describe how the models captured the structure of datasets [8, 20, 21, 23].”
Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis (2019, methodology): “The current limitation of these approaches is that there is a lack of empirical evidence to support the understanding by domain professionals [24].”
iForest: Interpreting Random Forests via Visual Analytics (2019, background): “[30] recently introduces another heatmap-based design to represent PDPs, in which the color encodes prediction probabilities.”
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference (2019, methodology): “One example is Prospector [48], which can be used to inves-Model-Understanding/Shaping Decision-Understanding/Shaping”