EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers

By Justin Talbot, Bongshin Lee, Ashish Kapoor, Desney S. Tan. 2009

272 citations recorded by the source corpus.

Machine learning is an increasingly used computational tool within human-computer interaction research. While most researchers currently utilize an iterative approach to refining classifier models and performance, we propose that ensemble classification techniques may be a viable and even preferable alternative. In ensemble learning, algorithms combine multiple classifiers to build one that is superior to its components. In this paper, we present EnsembleMatrix, an interactive visualization system that presents a graphical view of confusion matrices to help users understand relative merits of various classifiers. EnsembleMatrix allows users to directly interact with the visualizations in order to explore and build combination models. We evaluate the efficacy of the system and the approach in a user study. Results show that users are able to quickly combine multiple classifiers operating on multiple feature sets to produce an ensemble classifier with accuracy that approaches best-reported performance classifying images in the CalTech-101 dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Interactive Machine Learning via a GPU-accelerated Toolkit (2017, methodology): “EnsembleMatrix [35] uses custom visualizations to aid in the design of ensemble classifiers.”
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models (2018, background): “Talbot et al. [38] propose EnsembleMatrix that juxtaposes the confusion matrix of multiple classiﬁers for visual comparison and supports linear combinations of these classiﬁers for more effective model ensemble.”
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent (2012, background): “descriptions of the agent’s reasoning [13,15], visual depictions of the assistant’s known correct predictions versus its known failures [26], and electronic “door tags” displaying predictions of worker interruptibility with the reasons underlying each prediction (e.”
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning (2015, background): “Sometimes color or shading is also used to emphasize cell counts, drawing attention to problematic categories [22].”
Understanding Hidden Memories of Recurrent Neural Networks (2017, methodology): “These methods include: visual classiﬁcation [6,32,48], visual optimization [24,45], and visual feature engineering [4,41,49].”
Human model evaluation in interactive supervised learning (2011, background): “For example, ManiMatrix allows users to manipulate a confusion matrix to interactively steer a model’s performance to reflect their priorities [16], and EnsembleMatrix harnesses human users to optimize ensembles of learners [20].”
Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach (2011, unknown): “, [17, 18, 22, 30]), WYSIWYT/ML’s bug-finding complements them.”
Human-in-the-loop machine learning: a state of the art (2022, methodology): “Following this initial approach to IML we can cite the work of Talbot et al. (2009) in which the authors built a system called EnsembleMatrix , that was used to visualize the confusion matrices of various classifiers and allow users to interact with these visualizations to better understand the…”
Power to the People: The Role of Humans in Interactive Machine Learning (2014, methodology): “To facilitate the creation of ensemble classifiers, Talbot et al. (2009) developed EnsembleMatrix, a novel tool for helping people interactively build, evaluate, and explore different ensembles (Figure 8).”
Flock: Hybrid Crowd-Machine Learning Classifiers (2015, background): “Some systems help users choose between multiple machine learning models (e.g., [17]) and tune model parameters, for instance through visualizing confusion matrices [44].”
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models (2016, background): “[42]) allows users to combine already computed classification models to build ensemble models.”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “In constructing and refining a model, users require the ability to retrace steps in the event that recent actions have resulted in an undesired outcome (Talbot et al. 2009; Kapoor et al. 2010; Amershi et al. 2010).”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, background): “EnsembleMatrix [52] allows users to visually ensemble multiple classiﬁers together and provides a summary visualisation of results of these multiple classiﬁers.”
The What-If Tool: Interactive Probing of Machine Learning Models (2019, methodology): “En-sembleMatrix [31] is designed for comparing models that make up ensembles, while RuleMatrix [21] attempts to explain ML models in terms of simple rules.”