A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations

By Josua Krause, Aritra Dasgupta, Jordan Swartz, Yindalon Aphinyanagphongs, Enrico Bertini. 2017

100 citations recorded by the source corpus.

Human-in-the-loop data analysis applications necessitate greater transparency in machine learning models for experts to understand and trust their decisions. To this end, we propose a visual analytics workflow to help data scientists and domain experts explore, diagnose, and understand the decisions made by a binary classifier. The approach leverages “instance-level explanations”, measures of local feature relevance that explain single instances, and uses them to build a set of visual representations that guide the users in their investigation. The workflow is based on three main visual representations and steps: one based on aggregate statistics to see how data distributes across correct / incorrect decisions; one based on explanations to understand which features are used to make these decisions; and one based on raw data, to derive insights on potential root causes for the observed patterns. The workflow is derived from a long-term collaboration with a group of machine learning and healthcare professionals who used our method to make sense of machine learning models they developed. The case study from this collaboration demonstrates that the proposed workflow helps experts derive useful knowledge about the model and the phenomena it describes, thus experts can generate useful hypotheses on how a model can be improved.

Subfields: Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

ViCE: visual counterfactual explanations for machine learning models (2020, background): “[8] also provide a solution to a counterfactual question of how to change data to achieve a target class for black-box models.”
Manipulating and Measuring Model Interpretability (2018, background): “Threads of research that focus on this approach look at how to explain individual predictions by learning simple local approximations of a model around particular data points [63, 74, 89], training simple models to mimic more complex ones [64, 98], estimating the infuence of training data points [56”
The challenge of crafting intelligible intelligence (2018, background): “een proposed. Local Explanations: One way to simplify the explanation of a learned model is to make it relative to a single input query. Such explanations, which are termed local[33] or instance-based[20], are akin to a doctor explaining specific reasons for a patient’s diagnosis rather than communi”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, background): “…use visualizations for generating these explanations via rules [21], analogies, and layer activations [20], most approaches run into issues of visual complexity, the need to support multiple exploratory workﬂows, and reliance on simple, interpretable data for meaningful insights [17, 20, 21, 34].”
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models (2018, methodology): “Other research directions aim to identify important features and instances that are relevant to an issue within the model [7, 14–17].”
RuleMatrix: Visualizing and Understanding Classifiers with Rules (2018, background): “With the recent advances in machine learning, there is increasing need for transparent and interpretable machine learning models [8, 17, 32].”
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020, methodology): “Lei et al. (2016) use a simple Bernoulli distribution with REINFORCE (Williams, 1992) to overcome non-differentiability.”
A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning (2018, methodology): “[4] also proposed a visual analytics tool for the medical domain to debug binary classifiers with instance-level explanations.”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, methodology): “2017 [105] ◆ ◆ ◆”