High-Dimensional Probability Estimation with Deep Density Models

By Oren Rippel, Ryan P. Adams. 2013

128 citations recorded by the source corpus.

One of the fundamental problems in machine learning is the estimation of a probability distribution from data. Many techniques have been proposed to study the structure of data, most often building around the assumption that observations lie on a lower-dimensional manifold of high probability. It has been more difficult, however, to exploit this insight to build explicit, tractable density models for high-dimensional data. In this paper, we introduce the deep density model (DDM), a new approach to density estimation. We exploit insights from deep learning to construct a bijective map to a representation space, under which the transformation of the distribution of the data is approximately factorized and has identical and known marginal densities. The simplicity of the latent distribution under the model allows us to feasibly explore it, and the invertibility of the map to characterize contraction of measure across it. This enables us to compute normalized densities for out-of-sample data. This combination of tractability and flexibility allows us to tackle a variety of probabilistic tasks on high-dimensional datasets, including: rapid computation of normalized densities at test-time without evaluating a partition function; generation of samples without MCMC; and characterization of the joint entropy of the data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Density estimation using Real NVP (2016, methodology): “This formula has been discussed in several papers including the maximum likelihood formulation of independent components analysis (ICA) [4, 28], gaussianization [14, 11] and deep density models [5, 50, 17, 3].”
Variational Inference with Normalizing Flows (2015, methodology): “While it is straightforward to build invertible parametric functions for use in equation (5), e.g., invertible neural networks (Baird et al., 2005; Rippel & Adams, 2013), such approaches typically have a complexity for computing the Jacobian determinant that scales as O ( LD 3 ) , where D is the…”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “We demonstrate the utility of these diffusion probabilistic models by training high log likelihood models for a two-dimensional swiss roll, binary sequence, handwritten digit (MNIST), and several natural image (CIFAR-10, bark, and dead leaves) datasets.”
NICE: Non-linear Independent Components Estimation (2014, background): “(Rippel and Adams, 2013) reintroduces this idea but drops the bijectivity constraint but has to rely on a composite proxy to optimize the log-likelihood.”
Masked Autoregressive Flow for Density Estimation (2017, background): “Enforcing invertibility with nonsingular weight matrices has been proposed [1, 26], however in such approaches calculating the determinant of the Jacobian scales cubicly with data dimensionality in general.”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “The idea of composition saw its recent emergence in machine learning starting with Rippel and Adams (2013), who were perhaps the first to recognize that parameterizing flows with deep neural networks could result in quite general and expressive distribution classes.”
Avoiding pathologies in very deep networks (2014, neutral): “Deep Density Networks (Rippel and Adams, 2013) were constructed with invertibility in mind, with penalty terms encouraging the preservation of information about lower layers.”
Analyzing Inverse Problems with Invertible Neural Networks (2018, background): “Normalizing flows, introduced to the field of neural networks by [35, 34], have the potential to solve these problems.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “Differentiable bijective models were ﬁrst introduced to the deep learning community as likelihood-based generative models by Rippel & Adams (2013); Dinh et al. (2014), and as an inference machine by Rezende & Mohamed (2015).”
Early Visual Concept Learning with Unsupervised Deep Learning (2016, background): “Such constraints have been considered before [32, 29, 27], but no scalable unsupervised solution capable of disentangled factor learning based on these ideas yet exists.”
Block Neural Autoregressive Flow (2019, methodology): “In density estimation, for example, NFs are used to map observations from a complex (and unknown) data distribution to samples from a simple base distribution (Rippel and Adams, 2013).”
Emerging Convolutions for Generative Normalizing Flows (2019, methodology): “Generative models based on normalizing ﬂows (Rippel & Adams, 2013) have several advantages over other generative models: i) They optimize the log likelihood of a continuous distribution exactly, as opposed to Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) which…”
Relative gradient optimization of the Jacobian term in unsupervised deep learning (2020, methodology): “[44] Oren Rippel and Ryan Prescott Adams.”
A RAD approach to deep mixture models (2019, methodology): “The proposed approach will be a direct extension of ﬂow based models (Rippel & Adams, 2013; Dinh et al., 2015; 2017; Kingma & Dhariwal, 2018).”
Integer Discrete Flows and Lossless Compression (2019, methodology): “This work builds mainly upon flow-based generative models, described in [25, 7, 8].”