Score-Based Generative Modeling through Stochastic Differential Equations
By Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole. 2020
11,485 citations recorded by the source corpus.
Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Dacheng Tao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Chen Change Loy
- Andrea Vedaldi
In Statistical and Nonlinear Physics
- Arash Vahdat
- Stefano Ermon
- Karsten Kreis
- Yang Song
- Jonathan Ho
- Jun Zhu
- Diederik P. Kingma
- Jiaming Song
In Generative Models
- Arash Vahdat
- Stefano Ermon
- Karsten Kreis
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Jiaming Song
- Alexei A. Efros
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “An intriguing feature of diffusion models is that unconditional models can be conditioned at test-time [14, 77, 80].”
- Classifier-Free Diffusion Guidance (2022, methodology): “We train a diffusion model in continuous time [17, 2, 8]: letting z = {zλ |λ ∈ [λmin, λmax]} for hyperparameters λmin < λmax ∈ R, the forward process q(z|x) is the variance-preserving Markov process [15] specified as”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “Stap et al. (2020) propose a method for editing images of faces using feature vectors grounded in text. Bau et al. (2021) pair CLIP with state-of-the-art GAN models to inpaint images using text targets.”
- Generative Adversarial Networks (2021, neutral): “We have discussed the key design choices and variations in the GAN architecture, as well as the different LFs utilized to train GAN models.”
- Variational Diffusion Models (2021, methodology): “This would have been state-of-the-art until recently, but is worse than recent diffusion models that specifically target FID scores [Nichol and Dhariwal, 2021, Song et al., 2021b, Ho et al., 2020].”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “This can be implemented in many ways such as rejection sampling or Langevin dynamics [66, 86, 27].”
- Score-based Generative Modeling in Latent Space (2021, methodology): “[2] show how this modeling approach is described with a stochastic differential equation (SDE) framework which can be converted to maximum likelihood training [4].”
- On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “Song et al. (2020a) introduce denoising diffusion implicit models (DDIM), which contain non-Markovian processes that lead to an equivalent training objective as DDPM.”
- Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “There have been a number of improvements and alternatives proposed to the diffusion framework, for example generalization to continuous time [28], deterministic sampling [25], adversarial training [11], and others [6].”
- Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “• Rescaling residual connections with 1 √ 2 , following [53, 21, 22].”
- Image Super-Resolution via Iterative Refinement (2021, methodology): “The SR3 architecture is similar to the U-Net in DDPM [1], with self-attention and modifications adapted from [40]; i.”
- Noise Estimation for Generative Diffusion Models (2021, background): “A further development in the score based generative models is to consider this process as a solution to a stochastic differential equation [32].”
- Symbolic Music Generation with Diffusion Models (2021, methodology): “An exception to this is the recent work in energy-based and score-based generative modeling where the composition of energy functions and flexibility of Langevin dynamics has allowed for conditional sampling in the data space directly (Du & Mordatch, 2020; Song et al., 2020).”
- Denoising Diffusion Implicit Models (2020, methodology): “In fewer sampling steps, however, these choices will make a difference; we take Euler steps with respect to d σ ( t ) (which depends less directly on the scaling of “time” t ) whereas Song et al. (2020) take Euler steps with respect to d t .”
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, neutral): “…noise image, and takes multiple steps in input space that can be thought of as (noisy) gradient steps on an (implicit) energy function (Ho et al., 2020; Song et al., 2021a;b) Informally, one can think of these steps as progressively moving this initial random point closer to the ”data manifold” M .”
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Diffusion-based (DPMs) [22, 46] and score-based [49] generative models have recently succeeded in synthesizing realistic and high-resolution images, rivaling those from GANs [11, 15, 23].”
- Improved Denoising Diffusion Probabilistic Models (2021, methodology): “We did not attempt to calculate NLL using DDIM, since Song et al. (2020a) does not present NLL results or a simple way of estimating likelihood under DDIM.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “[66] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.”