Density estimation using Real NVP
By Laurent Dinh, Jascha Narain Sohl-Dickstein, Samy Bengio. 2016
4,503 citations recorded by the source corpus.
Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation and latent variable manipulations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Percy Liang
- Pieter Abbeel
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Stefano Ermon
- Arash Vahdat
- Yoshua Bengio
- Jiaming Song
- Diederik P. Kingma
- Jan Kautz
- Jun-Yan Zhu
- Aaron C. Courville
In Probabilistic & Bayesian Methods
- Stefano Ermon
- Yoshua Bengio
- Max Welling
- Diederik P. Kingma
- Yang Song
- Aaron C. Courville
- Arash Vahdat
- David M. Blei
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, background): “…(GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [12, 25, 3, 55, 35, 23, 10, 30, 41, 54, 24, 31, 42], and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable…”
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Variational autoencoders (VAE) [45] and flow-based models [18, 19] enable efficient synthesis of high resolution images [9, 43, 89], but sample quality is not on par with GANs.”
- Jukebox: A Generative Model for Music (2020, methodology): “…2017) use the chain rule of probability to factorize the joint distribution of data into a product of simpler distributions, and flow-based models (Dinh et al., 2015; 2017; Rezende & Mohamed, 2015; Kingma & Dhari-wal, 2018) learn a series of invertible transformations that maps the data…”
- Neural Discrete Representation Learning (2017, methodology): “Extensions include autoregressive prior and posterior models [14], normalising flows [31, 10], and inverse autoregressive posteriors [22].”
- Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “IAFs are a special type of normalising flow [3, 22, 4] which model a multivariate distribution p X ( x ) as an explicit invertible non-linear transformation f of a simple tractable distribution p Z ( z ) (such as an isotropic Gaussian distribution).”
- Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “, 2014), Flow based models (Dinh et al., 2017) and autoregressive models (Bengio & Bengio, 2000; Larochelle & Murray, 2011).”
- Stochastic Normalizing Flows (2020, background): “2 with any invertible differentiable transformation, such as NICE (Dinh et al., 2014), RealNVP (Dinh et al., 2016), invertible ResNets (Behrmann et al., 2018) or neural ODEs (Chen et al., 2018) – see (Papamakarios et al., 2019) for an overview of flow architectures.”
- Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Recently, flow-based models (henceforth simply called flow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
- Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “We use the same preprocessing as in [18].”
- Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “We distinguish two main types of generative models: likelihood based models, which include VAEs [16, 31], flow based [9, 30, 10, 17] and autoregressive models [20, 39]; and implicit generative ∗Equal contributions.”
- Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “We also show that the learned short-run MCMC can be used as a generative model, such as a generator model [13, 24] or the flow model [8, 9, 25, 4, 14], with the Langevin dynamics serving as a noiseinjected residual network, with the initial image serving as the latent variables, and with the initial”
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “The three main classes of powerful explicit density models are autoregressive models [26, 57], flowbased models [8, 9, 21, 16], and probabilistic latent variable models [24, 40, 33].”
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Another example, which we build upon in our work here, is the affine coupling layer (Dinh et al., 2016).”
- Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “In our case, we use an affine coupling layer [12].”
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “Examples include NICE (Dinh et al., 2014), Real NVP (Dinh et al., 2017), and Glow (Kingma & Dhariwal, 2018).”
- Glow: Generative Flow with Invertible 1x1 Convolutions (2018, methodology): “In Dinh et al. (2016), the authors propose the use of batch normalization (Ioffe and Szegedy, 2015) to alleviate the problems encountered when training deep models.”
- PixelSNAIL: An Improved Autoregressive Generative Model (2017, neutral): “Method CIFAR-10 ImageNet 32 × 32 Conv DRAW [4] 3.5 4.40 Real NVP [2] 3.49 4.28 VAE with IAF [5] 3.11 – PixelRNN [8] 3.00 3.86 PixelCNN [12] 3.03 3.83 Image Transformer [1] 2.98 3.81 PixelCNN++ [11] 2.92 – Block Sparse PixelCNN++ [9] 2.90 – PixelSNAIL (ours) 2.85 3.80”
- Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Recent work (Dinh et al., 2016), introduces RNVP, an invertible transformation that operates by, at each layer, modifying only a subset of the variables by a function that depends solely on the remaining variables.”
- PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “5.40 NICE (Dinh et al., 2014) 4.48 DRAW (Gregor et al., 2015) 4.13 Deep GMMs (van den Oord & Dambre, 2015) 4.00 Conv DRAW (Gregor et al., 2016) 3.58 Real NVP (Dinh et al., 2016) 3.49 PixelCNN (van den Oord et al., 2016b) 3.14 VAE with IAF (Kingma et al., 2016) 3.11 Gated PixelCNN (van den Oord et al”
- Improved Variational Inference with Inverse Autoregressive Flow (2016, background): “(van den Oord et al., 2016) 4.70 NICE (Dinh et al., 2014) 4.48 Deep GMMs (van den Oord and Schrauwen, 2014) 4.00 Real NVP (Dinh et al., 2016) 3.49 Pixel CNN van den Oord et al. (2016) 3.14 Pixel RNN (Row LSTM) van den Oord et al. (2016) 3.07 Pixel RNN (Diagonal BiLSTM) van den Oord et al. (2016)…”