Detecting Out-of-Distribution Inputs to Deep Generative Models Using a Test for Typicality

By Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Balaji Lakshminarayanan. 2019

85 citations recorded by the source corpus.

Recent work has shown that deep generative models can assign higher likelihood to out-of-distribution data sets than to their training data. We posit that this phenomenon is caused by a mismatch between the model's typical set and its areas of high probability density. In-distribution inputs should reside in the former but not necessarily in the latter, as previous work has presumed. To determine whether or not inputs reside in the typical set, we propose a statistically principled, easy-to-implement test using the empirical distribution of model likelihoods. The test is model agnostic and widely applicable, only requiring that the likelihood can be computed or closely approximated. We report experiments showing that our procedure can successfully detect the out-of-distribution sets in several of the challenging cases reported by Nalisnick et al. (2019).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “Our work builds heavily on Nijkamp et al. (2019b;a); Du & Mordatch (2019) which scales the training of EBMs to high-dimensional data using Contrastive Divergence and SGLD.”
Achieving Robustness in the Wild via Adversarial Mixing With Disentangled Representations (2019, background): “[47], due to concentration of measure, a generative model usually draws samples from its typical set [48] (a subset of the model’s full support) rather than regions of high probability density.”
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, background): “A long line of literature has thus been proposed, including density-based [74, 46, 6, 47, 11, 55, 61, 17], reconstruction-based [58, 76, 9, 54, 52, 7], one-class classifier [59, 56], and self-supervised [15, 25, 2] approaches.”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “High density and typical sets In prior work, poor performance of normalizing flows in out-ofdistribution detection is explained by the discrepancy between typical and high-density sets of a generative model [30].”
Input complexity and out-of-distribution detection with likelihood-based generative models (2019, methodology): “Choi et al. (2018) and Nalisnick et al. (2019b) suggest that typicality is the culprit for likelihood-based generative models not being able to detect OOD inputs.”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, methodology): “15 for the other experiment, and (3) the test for typicality by Nalisnick et al. (2019) ( TT ).”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “As an additional evaluation of the learned model’s performance, we consider two tasks using deep EBMs: the first one is out-of-distribution detection, where we follow previous work [6, 43] to use typicality as the detection metric (details in Appendix C.”
Identifying viruses from metagenomic data using deep learning (2020, methodology): “Various methods based on discriminative models or generative models have been developed for this purpose [53 – 60] and worth our future investigation.”
CutPaste: Self-Supervised Learning for Anomaly Detection and Localization (2021, background): “However, the anomaly score defined as an aggregation of pixel-wise reconstruction error or probability densities lacks to capture a high-level semantic information [42, 37].”