Do Deep Generative Models Know What They Don't Know?

By Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan. 2018

882 citations recorded by the source corpus.

A neural network deployed in the wild may be asked to make predictions for inputs that were drawn from a different distribution than that of the training data. A plethora of work has demonstrated that it is easy to find or synthesize inputs for which a neural network is highly confident yet wrong. Generative models are widely viewed to be robust to such mistaken confidence as modeling the density of the input features can be used to detect novel, out-of-distribution inputs. In this paper we challenge this assumption. We find that the density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former. Moreover, we find evidence of this phenomenon when pairing several popular image data sets: FashionMNIST vs MNIST, CelebA vs SVHN, ImageNet vs CIFAR-10 / CIFAR-100 / SVHN. To investigate this curious behavior, we focus analysis on flow-based generative models in particular since they are trained and evaluated via the exact marginal likelihood. We find such behavior persists even when we restrict the flows to constant-volume transformations. These transformations admit some theoretical analysis, and we show that the difference in likelihoods can be explained by the location and variances of the data and the model curvature. Our results caution against using the density estimates from deep generative models to identify inputs similar to the training distribution until their behavior for out-of-distribution inputs is better understood.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Safety, Robustness & Interpretability

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “Nalisnick et al. (2018) showed that tractable deep generative models such as Kingma & Dhariwal (2018) and Salimans et al. (2017) can assign higher densities to OOD examples than in-distribution examples.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “Following Nalisnick et al. (2019), we use area under the ROC curve (AUROC) as quantitative metric, where high AUROC indicates that the model correctly assigns low density to OOD samples.”
Continual Learning via Local Module Composition (2021, background): “Several recent works have shown that deep generative models often mistakenly assign high likelihood values to outlier points [68, 36, 95].”
Posterior Meta-Replay for Continual Learning (2021, methodology): “While in practice this approach still seems to be out of reach for high-dimensional problems (Nalisnick et al., 2019), generative models for data replay have been successfully applied to CL (Shin et al.”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Even if we use a hybrid model (Nalisnick et al., 2019b), then the discriminative part, a softmax/cross entropy model, does not have reliable predictive uncertainty.”
Towards neural networks that provably know when they don't know (2019, background): “Moreover, it has recently been discovered that VAEs, ﬂows and GANs also suffer from overconﬁdent likelihoods (Nalisnick et al., 2019; Hendrycks et al., 2019) far away from the data they are supposed to model as well as adversarial samples (Kos et al., 2017).”
Out-of-Domain Detection for Natural Language Understanding in Dialog Systems (2019, background): “Popular approaches include modeling the probability density [8], [25], computing reconstruction losses [7], [26],”
Deep Anomaly Detection with Outlier Exposure (2018, background): “Anomalous examples should have low probability density, as they are scarce in Din by definition (Nalisnick et al., 2019).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “This is to provide an intuitive picture of the challenge of open set recognition with neural networks and to summarize and repeat the ﬁndings of the much more detailed experiments presented in numerous prior works (Mundt et al., 2022, 2019; Nalisnick et al., 2019; Ovadia et al., 2019).”
Conditioning by adaptive sampling for robust design (2019, background): “Recently some generative models have themselves been shown to exhibit pathologies (Nalisnick et al., 2019), however we have not observed any such phenomenon in our setting.”
Energy-based Out-of-distribution Detection (2020, background): “[28], deep generative models can assign a high likelihood to out-of-distribution data.”
Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “We provide histograms of relative likelihoods for SVHN in Figure 10 which are also discussed in (Nalisnick et al., 2019; Hendrycks et al., 2018).”
Likelihood Ratios for Out-of-Distribution Detection (2019, methodology): “…Glow models trained on the CIFAR-10 image dataset assign higher likelihood to OOD inputs from the SVHN dataset than they do to in-distribution CIFAR-10 inputs; Nalisnick et al. (2018), Shafaei et al. (2018) and Hendrycks et al. (2018) show failure modes of PixelCNN and PixelCNN++ for OOD detection.”