Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
By Himabindu Lakkaraju, Ece Kamar, R. Caruana, Eric Horvitz. 2016
160 citations recorded by the source corpus.
Predictive models deployed in the real world may assign incorrect labels to instances with high confidence. Such errors or unknown unknowns are rooted in model incompleteness, and typically arise because of the mismatch between training data and the cases encountered at test time. As the models are blind to such errors, input from an oracle is needed to identify these failures. In this paper, we formulate and address the problem of informed discovery of unknown unknowns of any given predictive model where unknown unknowns occur due to systematic biases in the training data.We propose a model-agnostic methodology which uses feedback from an oracle to both identify unknown unknowns and to intelligently guide the discovery. We employ a two-phase approach which first organizes the data into multiple partitions based on the feature similarity of instances and the confidence scores assigned by the predictive model, and then utilizes an explore-exploit strategy for discovering unknown unknowns across these partitions. We demonstrate the efficacy of our framework by varying the underlying causes of unknown unknowns across various applications. To the best of our knowledge, this paper presents the first algorithmic approach to the problem of discovering unknown unknowns of predictive models.
Subfields: Artificial Intelligence, Reinforcement Learning, Anomaly & Outlier Detection
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Song-Chun Zhu
- Percy Liang
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
In Reinforcement Learning
- Enrique Munoz de Cote
- Josh Tenenbaum
- Jacob William Crandall
- Iyad Rahwan
- Pablo Hernandez-Leal
- Michael Kaisers
- Miroslav Dud'ik
- Dawn Song
In Anomaly & Outlier Detection
Citations in the corpus
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, neutral): “For example, researchers have used these high-dimensional “error terrains” to understand the effects of demographic factors, sociocultural factors, and environmental factors on face-based AI systems (Lakkaraju et al. 2017; Nushi, Kamar, and Horvitz 2018).”
- Interpreting CNNs via Decision Trees (2018, background): “[19] discovered knowledge blind spots (unknown patterns) of a pre-trained CNN in a weaklysupervised manner.”
- Interpretable Convolutional Neural Networks (2017, background): “ple to manually check image regions accountable for the label prediction for each testing image. [9] extracted relationships between representations of various categories from a CNN. Lakkaraju et al. [14] and Zhang et al. [37] explored unknown knowledge of CNNs via active annotations and active ques”
- Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “the-loop approaches to fairness auditing that combine the strengths of automated methods and human inspection (cf. [24, 53, 100, 106]), perhaps aided by tools for visualization and guided exploration [24, 69]. Finally, at a broader, organizational level, future research should explore how teams and ”
- Examining CNN representations with respect to Dataset Bias (2017, background): “Lakkaraju et al. (Lakkaraju et al. 2017) and Zhang et al. (Zhang et al. 2017b; 2017a) explored unknown knowledge hidden in CNNs via active annotations and active question-answering.”
- Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, background): “Later work has proposed data collection algorithms for addressing overconfident predictions (Lakkaraju et al. 2017; Bansal and Weld 2018).”
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “…(RL), MAS and multi-armed bandits (to name a few) have all acknowledged the fact that specialized targeted work is needed that explicitly addresses non-stationary environments (Sutton et al., 2007; Panait and Luke, 2005; Garivier and Moulines, 2011; Matignon et al., 2012; Lakkaraju et al., 2017).”
- On Interpretability of Artificial Neural Networks: A Survey (2020, background): “To recognize those mislabeled instances with high predictive scores in the dataset efficiently, H. Lakkaraju et al. [97] introduced two basic speculations: The first is that mislabeling an instance with high confidence is due to the systematic biases instead of random perturbation, while the second…”
- Machine behaviour (2019, background): “The dimensionality and size of data add another layer of complexity to understanding machine behaviour 49 .”
- Anomalous Instance Detection in Deep Learning: A Survey (2020, result): “This is similar to the concept of unknown-unknowns [108].”
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “s: (1) in high confidence regions (circles in Figure5), not only did workers have to agree more, but they also had to identify cases where the model failed with very high confidence (unknown unknowns [37]). Identifying unknown unknowns could have been a difficult and time-consuming task for workers,”
- Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation (2020, background): “categories according to the corresponding interpreting results: visualizing pre-trained models[10][11], diagnose pre-trained models[12][13] and construct interpretable models[14][15].”
- IncluSet: A Data Surfacing Repository for Accessibility Datasets (2020, background): “of real-world phenomena, and exclusion [18]– issues of fairness pertaining to underrepresented populations.”
- Understanding the Effect of Accuracy on Trust in Machine Learning Models (2019, background): “However, a model’s stated performance may not accurately reflect its performance post deployment because the data on which the model was trained and evaluated may look very different from real-world use cases [15].”
- Explanatory Dialogs: Towards Actionable, Interactive Explanations (2018, background): “Previous work by Lakkaraju et al. (2017) on identifying unknown unknowns assumes that the utility of each revealed UU is independent of the others, rather than considering the set holistically.”