Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models

By Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, David S. Ebert. 2018

221 citations recorded by the source corpus.

Interpretation and diagnosis of machine learning models have gained renewed interest in recent years with breakthroughs in new approaches. We present Manifold, a framework that utilizes visual analysis techniques to support interpretation, debugging, and comparison of machine learning models in a more transparent and interactive manner. Conventional techniques usually focus on visualizing the internal logic of a specific model type (i.e., deep neural networks), lacking the ability to extend to a more complex scenario where different model types are integrated. To this end, Manifold is designed as a generic framework that does not rely on or access the internal logic of the model and solely observes the input (i.e., instances or features) and the output (i.e., the predicted result and probability distribution). We describe the workflow of Manifold as an iterative process consisting of three major phases that are commonly involved in the model development and diagnosis process: inspection (hypothesis), explanation (reasoning), and refinement (verification). The visual components supporting these tasks include a scatterplot-based visual summary that overviews the models' outcome and a customizable tabular view that reveals feature discrimination. We demonstrate current applications of the framework on the classification and regression tasks and discuss other potential machine learning use scenarios where Manifold can be applied.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

FairSight: Visual Analytics for Fairness in Decision Making (2019, methodology): “Mainfold [42] suggested a model-agnostic framework to interpret the outcome, inspect a subset of instances, and reﬁne the feature set or model, to facilitate the comparison of models.”
ViCE: visual counterfactual explanations for machine learning models (2020, methodology): “For example, Manifold [24] enables the comparison of data distribution at different levels of granularity; RuleMatrix [14] visualizes extracted rules for a given model; iForest [25] and EnsembleMatrix [19] attempt to explain ensemble models.”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, background): “expected system behavior, and downstream performance regression [2, 32]. Such issues have also been highlighted in surveys on software engineering challenges in ML [1, 42]. New performance evaluation [7, 31, 41] approaches suggest reporting performance on data slices rather than aggregate measures t”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “…to the benchmarking paradigm, such as test suites [1, 126, 170], for example, can re-orient ML evaluation away from its current focus on the competitive determination of “state of the art”, and more towards an exploratory and descriptive probing of model capabilities [65, 106, 143, 161, 169].”
Tempura: Query Analysis with Structural Templates (2020, background): “First, developers may start by looking at aggregate performance metrics like model accuracy, and then drill down into more actionable groups [39, 44].”
DeepVID: Deep Visual Interpretation and Diagnosis for Image Classifiers via Knowledge Distillation (2019, unknown): “Localized Model-Agnostic Interpretation [13], [14].”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “The first ameliorates the unavoidable randomness of finite evaluations while the second enables error analysis (Wu et al., 2019) and model probing (Belinkov and Glass, 2019; Zhang et al., 2019).”
Robustness Gym: Unifying the NLP Evaluation Landscape (2021, background): “, 2019), Manifold (Zhang et al., 2018), Pytorch Captum (Narine Kokhlikyan and Reblitz-Richardson), DiCE (Mothilal et al.”
Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations (2020, background): “, 2017) as well as to compare models with each other (Zhang et al., 2018).”
Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “The tool is not production ready at the time of writing, but a prototype has been developed in one of the most recently published papers by their research team [40].”
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference (2019, background): “Similar debugging-focused approaches [47,91] only visualize the necessary parts of a model to ﬁnd and explain errors, instead of giving in-depth insights into hidden model states.”
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020, background): “Many are general-purpose, such as the What-If Tool (Wexler et al., 2020), Cap-tum (Kokhlikyan et al., 2019), Manifold (Zhang et al., 2018), or InterpretML (Nori et al., 2019), while others focus on speciﬁc applications like fairness, including FairVis (Cabrera et al., 2019) and FairSight (Ahn and Li”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “Manifold [181] and MLCube Explorer [182] are two visual analytical tools that provide comparative analysis for multiple models.”
Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models (2019, methodology): “Example tasks include interactive model debugging and performance analysis [2, 44, 53], feature ideation and selection [6, 34], instance subset inspection and comparison [30, 31], model comparison [67], and constructing interpretable and trustworthy visual analysis systems [11].”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, methodology): “…use visualizations for generating these explanations via rules [21], analogies, and layer activations [20], most approaches run into issues of visual complexity, the need to support multiple exploratory workﬂows, and reliance on simple, interpretable data for meaningful insights [17, 20, 21, 34].”