Deep Unsupervised Learning using Nonequilibrium Thermodynamics
By Jascha Narain Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli. 2015
10,395 citations recorded by the source corpus.
A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Ruslan Salakhutdinov
- Trevor Darrell
In Statistical and Nonlinear Physics
- Arash Vahdat
- Stefano Ermon
- Yang Song
- Jonathan Ho
- Karsten Kreis
- Jiaming Song
- Diederik P. Kingma
- Prafulla Dhariwal
In Generative Models
- Stefano Ermon
- Arash Vahdat
- Jun-Yan Zhu
- Jiaming Song
- Richard Zhang
- Karsten Kreis
- Yoshua Bengio
- Eli Shechtman
In Probabilistic & Bayesian Methods
- Stefano Ermon
- Yoshua Bengio
- Yang Song
- Jiaming Song
- Diederik P. Kingma
- Arash Vahdat
- Aaron C. Courville
- Max Welling
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, methodology): “We set T = 1000 for all experiments so that the number of neural network evaluations needed during sampling matches previous work [50, 52].”
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Recently, Diffusion Probabilistic Models (DM) [79], have achieved state-of-the-art results in density estimation [44] as well as in sample quality [15].”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Concurrently, diffusion models [46, 48, 25] have emerged as a promising generative modeling framework, pushing the state-of-the-art on image and video generation tasks [11, 26, 24].”
- Pixel Recurrent Neural Networks (2016, neutral): “, 2014), [2] (Sohl-Dickstein et al., 2015), [3] (van den Oord & Schrauwen, 2014a).”
- Stochastic Normalizing Flows (2020, background): “More closely related is (Sohl-Dickstein et al., 2015) which uses of stochastic flows for density estimation and trains diffusion kernels by maximizing a variational bound of the data likelihood.”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “Nonequilibrium Thermodynamics (NET [51]) used a prescribed diffusion process to slowly transform data into random noise, and then learned to reverse this procedure by training an inverse diffusion.”
- Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Recently, probabilistic models parameterized by deep neural networks have achieved great success at approximately sampling from highly complex, multi-modal empirical distributions [11, 6, 3, 20].”
- Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “, 2013b), and NET approaches (Sohl-Dickstein et al., 2015) but it retains two crucial differences.”
- Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “Drawing on both Sohl-Dickstein et al. (2015) and the walkback procedure introduced for GSN in Alain et al. (2016), a variational variant of the walkback algorithm was investigated by Goyal et al. (2017) at the same time as our work.”
- PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, background): “Model Bits per sub-pixel Deep Diffusion (Sohl-Dickstein et al., 2015) 5.”
- Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “(van den Oord et al., 2016) 4.70 NICE (Dinh et al., 2014) 4.48 Deep GMMs (van den Oord and Schrauwen, 2014) 4.00 Real NVP (Dinh et al., 2016) 3.49 Pixel CNN van den Oord et al. (2016) 3.14 Pixel RNN (Row LSTM) van den Oord et al. (2016) 3.07 Pixel RNN (Diagonal BiLSTM) van den Oord et al. (2016) 3.0”
- Density estimation using Real NVP (2016, background): “Still, the approximation in the inference process limits its ability to learn high dimensional deep representations, motivating recent work in improving approximate inference [42, 48, 55, 63, 10, 59, 34].”
- Towards Conceptual Compression (2016, neutral): “, 2014), [2] (Sohl-Dickstein et al., 2015), [3] (van den Oord & Schrauwen, 2014), [4] (van den Oord et al.”
- GSNs : Generative Stochastic Networks (2015, background): “Another approach that requires a sequence of sampling steps and that is maybe more related to DAEs and GSNs, is the ‘non-equilibrium thermodynamics’ approach of Sohl-Dickstein et al. (2015).”
- Classifier-Free Diffusion Guidance (2022, methodology): “We train a diffusion model in continuous time [17, 2, 8]: letting z = {zλ |λ ∈ [λmin, λmax]} for hyperparameters λmin < λmax ∈ R, the forward process q(z|x) is the variance-preserving Markov process [15] specified as”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “…for post-generation image filtering (utilizing CLIP [44] for instance), we extend the use of classifier-free guidance suggested for diffusion models [53, 20] by [22, 41] to transformers, eliminating the need for post-generation filtering, thus producing faster and higher quality generation results,…”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “We consider the Gaussian diffusion models introduced by Sohl-Dickstein et al. (2015) and improved by Song & Ermon (2020b); Ho et al. (2020).”
- Variational Diffusion Models (2021, methodology): “Our work builds on diffusion probabilistic models (DPMs) [Sohl-Dickstein et al., 2015], or diffusion models in short.”