Generative Modeling by Estimating Gradients of the Data Distribution
By Yang Song, Stefano Ermon. 2019
5,774 citations recorded by the source corpus.
We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.
Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Dacheng Tao
In Biophysics
- Prafulla Dhariwal
- Yang Song
- Jonathan Ho
- Diederik P. Kingma
- Jong Chul Ye
- Richard Zhang
- Alex Nichol
- Arash Vahdat
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Arash Vahdat
- Stefano Ermon
- Karsten Kreis
- Jun-Yan Zhu
- Jiaming Song
- Richard Zhang
- Eli Shechtman
- Jan Kautz
In Probabilistic & Bayesian Methods
- Stefano Ermon
- Yoshua Bengio
- Yang Song
- Arash Vahdat
- Diederik P. Kingma
- Jiaming Song
- Aaron C. Courville
- Yilun Du
Citations in the corpus
- Denoising Diffusion Probabilistic Models (2020, methodology): “…flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [12, 25, 3, 55, 35, 23, 10, 30, 41, 54, 24, 31, 42], and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable to those of GANs [11, 52].”
- Improved Techniques for Training Score-Based Generative Models (2020, methodology): “[9] noticed that adding an extra denoising step after the original annealed Langevin dynamics in [1], similar to [10, 11, 12], often significantly improves FID scores [13] without affecting the visual appearance of samples.”
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “This is similar to the objective of Score Matching (Hyvärinen, 2005) which can also be used to train EBMs (Kingma & Lecun, 2010; Song & Ermon, 2019).”
- Flow Contrastive Estimation of Energy-Based Models (2019, background): “Another line of work is to estimate the parameters of EBM by score matching [26, 74, 63, 64].”
- Classifier-Free Diffusion Guidance (2022, background): “This objective is denoising score matching [18] over multiple noise scales [16], and when p(λ) is uniform, the objective is proportional to the variational lower bound on the marginal log likelihood of the latent variable model ∫ pθ(x|z)pθ(z)dz, ignoring the term for the unspecified pθ(x|z) and for ”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “We consider the Gaussian diffusion models introduced by Sohl-Dickstein et al. (2015) and improved by Song & Ermon (2020b); Ho et al. (2020).”
- Variational Diffusion Models (2021, methodology): “Another special case is the variance-exploding diffusion process as used by [Song and Ermon, 2019, Song et al., 2021b], where ↵2t = 1.”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Most LVGMs are built on top of four paradigms: variational autoencoders (VAEs, [52, 77]), Normalizing Flows (NFs, [28, 29]), Generative Adversarial Networks (GANs, [34]), and diffusion / score-based generative models [42, 85].”
- Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
- On Fast Sampling of Diffusion Probabilistic Models (2021, background): “They retrain the network conditioned on continuous noise levels instead of discrete diffusion steps (Song and Ermon, 2019; Chen et al., 2020; Okamoto et al., 2021; San-Roman et al., 2021).”
- Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “It is a weighted form of the ELBO that resembles denoising score matching over multiple noise scales [10, 26].”
- Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “Diffusion models are a class of likelihood-based models which have recently been shown to produce high-quality images [49, 52, 19] while offering desirable properties such as distribution coverage, a stationary training objective, and easy scalability.”
- VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…al., 2017; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Ho et al., 2019a), Oord et al., 2017; Razavi et al., 2019; Ramesh et al., 2021), and lately diffusion and score matching models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020).”
- Image Super-Resolution via Iterative Refinement (2021, background): “Thus, the denoising model is aware of the level of noise through conditioning on g, similar to [24], [33].”
- Noise Estimation for Generative Diffusion Models (2021, methodology): “In [31], a new generative model based on the score matching method [10] and Langevin dynamics was introduced.”
- Symbolic Music Generation with Diffusion Models (2021, methodology): “The following objective is simpler to train, resembles denoising score matching [5, 14] and was found to yield higher-quality samples:”
- Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “The corrector algorithms We take the schedule of annealed Langevin dynamics in Song & Ermon (2019), but re-frame it with slight modifications in order to get better interpretability and empirical performance.”