Improved Techniques for Training Score-Based Generative Models

By Yang Song, Stefano Ermon. 2020

1,493 citations recorded by the source corpus.

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, background): “Our (cid:15) -prediction reverse process parameterization establishes a connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics [52, 53].”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “We use diffusion models [25, 48] to produce images conditioned on CLIP image embeddings (and optionally text captions).”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “(2019); Tao et al. (2020); Zhang et al. (2021); Ye et al. (2021) train GANs with text-conditioning using publicly available image captioning datasets. Ramesh et al. (2021) synthesize images conditioned on text by building on the approach of van den Oord et al. (2017), wherein an autoregressive gener”
Variational Diffusion Models (2021, background): “Interest in diffusion models has recently reignited due to their impressive image generation results [Ho et al., 2020, Song and Ermon, 2020].”
Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
On Fast Sampling of Diffusion Probabilistic Models (2021, positive): “Diffusion models are connected with scored-based models (Song and Ermon, 2019, 2020; Song et al., 2020b), and there have been a series of research extending and improving diffusion models (Song et al., 2020b; Gao et al., 2020; Dhariwal and Nichol, 2021; San-Roman et al., 2021; Meng et al., 2021).”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “Diffusion models [24] have recently been shown to synthesize high quality images and audio [3, 10, 14, 27], which have long been led by other classes of generative models such as autoregressive models, GANs, VAEs, and flows [2, 5, 7, 9, 12, 13, 18, 29, 30, 32].”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “They also note that training with this objective and using their corresponding sampling procedure is equivalent to the denoising score matching model from Song and Ermon [51], who use Langevin dynamics to sample from a denoising model trained with multiple noise levels to produce high quality image ”
Image Super-Resolution via Iterative Refinement (2021, background): “Song and Ermon [39] advocate the use of a Multi-scale Guassian mixture as the target density, where each data point is perturbed with different amounts of Gaussian noise, so that Langevin dynamics starting from pure noise can still yield reasonable samples.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “01 and σ max is chosen according to Technique 1 in Song & Ermon (2020).”
Denoising Diffusion Implicit Models (2020, methodology): “In fewer sampling steps, however, these choices will make a difference; we take Euler steps with respect to d σ ( t ) (which depends less directly on the scaling of “time” t ) whereas Song et al. (2020) take Euler steps with respect to d t .”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “This parameterization is motivated by its connection to denoising score matching with Langevin dynamics (Song & Ermon, 2019; 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “WaveG-rad estimates the gradients of the data density, leveraging on the diffusion probabilistic model (Ho et al., 2020) and score matching framework (Song et al., 2019b; Song & Ermon, 2020).”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “…considered other conditional generative approaches (we discussed some of the alternatives in section 2), the choice of the reverse diffusion approach (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; 2020; Song et al., 2021a;b; Nichol & Dhariwal, 2021) is not entirely arbitrary.”
Improved Denoising Diffusion Probabilistic Models (2021, background): “More recently, Ho et al. (2020) showed an equivalence between denoising diffusion probabilistic models (DDPM) and score based generative models (Song & Ermon, 2019; 2020), which learn a gradient of the log-density of the data distribution using denoising score matching (Hyv¨arinen, 2005).”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “As a state-of-the-art generative model family, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song and Ermon, 2020) progressively add noise to the training data and then learn the reverse process to generate samples.”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “Until recently, score-based and denoising diffusion probabilistic models were formalized in terms of Markov chains (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song & Ermon, 2020).”
Adversarial score matching and improved sampling for image generation (2020, methodology): “As we will show, the FIDs from Song and Ermon (2020) suffer from such distance.”