Hybrid Models with Deep and Invertible Features

By Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan. 2019

105 citations recorded by the source corpus.

We propose a neural hybrid model consisting of a linear model defined on a set of features computed by a deep, invertible transformation (i.e. a normalizing flow). An attractive property of our model is that both p(features), the density of the features, and p(targets | features), the predictive distribution, can be computed exactly in a single feed-forward pass. We show that our hybrid model, despite the invertibility constraints, achieves similar accuracy to purely predictive models. Moreover the generative component remains a good model of the input features despite the hybrid optimization objective. This offers additional capabilities such as detection of out-of-distribution inputs and enabling semi-supervised learning. The availability of the exact joint density p(targets, features) also allows us to compute many quantities readily, making our hybrid model a useful building block for downstream applications of probabilistic deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Even if we use a hybrid model (Nalisnick et al., 2019b), then the discriminative part, a softmax/cross entropy model, does not have reliable predictive uncertainty.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Methods which model the joint distribution p ( y, x ) , e.g. deep hybrid models (Kingma et al., 2014; Alemi et al., 2018; Nalisnick et al., 2019; Behrmann et al., 2018).”
Do Deep Generative Models Know What They Don't Know? (2018, background): “[2] Eric Nalisnick, Akihiro Matsukawa, YeeWhye Teh, Dilan Gorur, and Balaji Lakshminarayanan.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Flows can be used for joint generative and predictive modeling by using them as the core component of a hybrid model (Nalisnick et al., 2019).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, background): “, 2019), semi-supervised learning (Nalisnick et al., 2019), anomaly detection (Choi et al.”
Likelihood Ratios for Out-of-Distribution Detection (2019, background): “A popular and intuitive strategy for detecting OOD inputs is to train a generative model (or a hybrid model cf. Nalisnick et al. (2019)) on training data and use that to detect OOD inputs at test time (Bishop, 1994).”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, background): “We hypothesize this can potentially improve the representation learned by an invertible model, for example in a semi-supervised setting (Nalisnick et al., 2019; Atanov et al., 2019) or as a component of a reversible model for memory-efficient backpropagation (Gomez et al.”
Contrastive Training for Improved Out-of-Distribution Detection (2020, background): “While generative models are a promising avenue for OOD detection, applying them directly to the image space has not achieved state of the art results, even when combined with a classification network [26].”
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, background): “A popular and intuitive strategy for detecting outliers is to train a generative model[56], [57] to approximate the density distribution of training datasets.”
Semi-Supervised Learning with Normalizing Flows (2019, methodology): “FlowGMM outperforms the M1+M2 model and performs better or on par with DIGLM.”
Further Analysis of Outlier Detection with Deep Generative Models (2020, unknown): “Building on these concerns, various diagnostics have been deployed to evaluate the calibration of newly proposed DGMs [5–9], or applied when revisiting older modeling practices [10].”
Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Even if we use a hybrid model (Nalisnick et al., 2019b), then the discriminative part, a softmax/cross entropy model, does not have reliable predictive uncertainty.”
Densely connected normalizing flows (2021, methodology): “Still, modeling pθ(x) using bijective formulation enables exact likelihood evaluation and efficient sample generation, which makes this approach convenient for various downstream tasks [21, 22, 23].”
Understanding the Limitations of Conditional Generative Models (2019, background): “…attacks (Schott et al., 2019; Ghosh et al., 2019; Song et al., 2018; Li et al., 2018; Frosst et al., 2018), to enable robust classiﬁcation in the presence of outliers (Nalisnick et al., 2019b) and to achieve promising results in semi-supervised learning (Kingma et al., 2014; Salimans et al., 2016).”
Exploring the Limits of Out-of-Distribution Detection (2021, methodology): “There has been a lot of progress in detecting OOD inputs including methods based on discriminative models [Hendrycks and Gimpel, 2016, Lee et al., 2018, Liang et al., 2017, Liu et al., 2020] as well as methods based on deep generative models [Nalisnick et al., 2019, Zhang et al., 2020].”
Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic and Aleatoric Uncertainty (2021, background): “, CIFAR-10 vs SVHN and CIFAR-10 vs CIFAR-100) are known to be difficult [Nalisnick et al., 2019].”
Task-agnostic Continual Learning with Hybrid Probabilistic Models (2021, unknown): “Nalisnick et al. [2019b] and Kingma et al. [2014] provide another two examples of hybrid models for semi-supervised learning.”