NICE: Non-linear Independent Components Estimation

By Laurent Dinh, David Krueger, Yoshua Bengio. 2014

2,631 citations recorded by the source corpus.

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is one in which the data has a distribution that is easy to model. For this purpose, a non-linear deterministic transformation of the data is learned that maps it to a latent space so as to make the transformed data conform to a factorized distribution, i.e., resulting in independent latent variables. We parametrize this transformation so that computing the Jacobian determinant and inverse transform is trivial, yet we maintain the ability to learn complex non-linear transformations, via a composition of simple building blocks, each based on a deep neural network. The training criterion is simply the exact log-likelihood, which is tractable. Unbiased ancestral sampling is also easy. We show that this approach yields good generative models on four image datasets and can be used for inpainting.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

In Generative Models

In Signal Processing

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble ﬂows [9, 43, 10, 30, 5, 14, 21] and VAEs [31, 44, 34], diffusion models are designed so that q has no parameters and the top-level latent x T has nearly zero mutual information with the data x 0 .”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Variational autoencoders (VAE) [45] and flow-based models [18, 19] enable efficient synthesis of high resolution images [9, 43, 89], but sample quality is not on par with GANs.”
MADE: Masked Autoencoder for Distribution Estimation (2015, methodology): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et al., 2014).”
Jukebox: A Generative Model for Music (2020, methodology): “, 2017) use the chain rule of probability to factorize the joint distribution of data into a product of simpler distributions, and flow-based models (Dinh et al., 2015; 2017; Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018) learn a series of invertible transformations that maps the data distributio”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “IAFs are a special type of normalising ﬂow [3, 22, 4] which model a multivariate distribution p X ( x ) as an explicit invertible non-linear transformation f of a simple tractable distribution p Z ( z ) (such as an isotropic Gaussian distribution).”
Stochastic Normalizing Flows (2020, methodology): “2 with any invertible differentiable transformation, such as NICE (Dinh et al., 2014), RealNVP (Dinh et al.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Recently, ﬂow-based models (henceforth simply called ﬂow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “Recent progress is mainly driven by two approaches: likelihood-based methods [17, 29, 11, 60] and generative adversarial networks (GAN [15]).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “We distinguish two main types of generative models: likelihood based models, which include VAEs [16, 31], flow based [9, 30, 10, 17] and autoregressive models [20, 39]; and implicit generative ∗Equal contributions.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “We also show that the learned short-run MCMC can be used as a generative model, such as a generator model [13, 24] or the flow model [8, 9, 25, 4, 14], with the Langevin dynamics serving as a noiseinjected residual network, with the initial image serving as the latent variables, and with the initial”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “92 Without autoregressive components NICE [8] = 4 .”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “The key idea in the de-sign of a ﬂow model is to form f by stacking individual simple invertible transformations (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Rezende & Mohamed, 2015; Kingma et al., 2016; Louizos & Welling, 2017).”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, background): “Flow-based networks [11, 12, 1] solve this problem by ensuring the neural network mapping is invertible.”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “Partitioning the dimensions and using afﬁne transformations makes the determinant of the Jacobian cheap to compute, and the inverse f − 1 computable with the same cost as f (Dinh et al., 2014; 2017).”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, methodology): “This work builds upon the ideas and ﬂows proposed in (Dinh et al., 2014) (NICE) and (Dinh et al., 2016) (RealNVP); comparisons with this work are made throughout this paper.”
Neural Ordinary Differential Equations (2018, neutral): “The discretized equation (1) also appears in normalizing flows (Rezende and Mohamed, 2015) and the NICE framework (Dinh et al., 2014).”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, methodology): “These models (Dinh et al., 2014; Rezende & Mohamed, 2015; Dinh et al., 2016) typically apply an invertible transformation that admits tractable determinant calculation to some continuous entropy source.”
A-NICE-MC: Adversarial Training for MCMC (2017, methodology): “Fortunately, we can design such an proposal by using a volume preserving flow model [14].”
Learning to Generate Samples from Noise through Infusion Training (2017, background): “…thermodynamic slow diffusion process (Sohl-Dickstein et al., 2015) • Continuous transformation of a distribution by invertible functions (Dinh et al. (2014), also used for variational inference in Rezende & Mohamed (2015)) Several of these approaches are based on maximizing an explicit…”