Adversarial score matching and improved sampling for image generation

By Alexia Jolicoeur-Martineau, Remi Piche-Taillefer, Rémi Tachet des Combes, Ioannis Mitliagkas. 2020

138 citations recorded by the source corpus.

Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling. The approach works by first training a neural network to estimate the score of a distribution, and then using Langevin dynamics to sample from the data distribution assumed by the score network. Despite the convincing visual quality of samples, this method appears to perform worse than Generative Adversarial Networks (GANs) under the Frechet Inception Distance, a standard metric for generative models. We show that this apparent gap vanishes when denoising the final Langevin samples using the score network. In addition, we propose two improvements to DSM-ALS: 1) Consistent Annealed Sampling as a more stable alternative to Annealed Langevin Sampling, and 2) a hybrid training formulation, composed of both Denoising Score Matching and adversarial objectives. By combining these two techniques and exploring different network architectures, we elevate score matching methods and obtain results competitive with state-of-the-art image generation on CIFAR-10.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, background): “[9] noticed that adding an extra denoising step after the original annealed Langevin dynamics in [1], similar to [10, 11, 12], often significantly improves FID scores [13] without affecting the visual appearance of samples.”
Score-based Generative Modeling in Latent Space (2021, background): “[39] add an adversarial objective, [5] introduce non-Markovian diffusion processes that allow to trade off synthesis speed, quality, and sample diversity, [40] learn a sequence of conditional energy-based models for denoising, [41] distill the iterative sampling process into single shot synthesis, a”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “There have been a number of improvements and alternatives proposed to the diffusion framework, for example generalization to continuous time [28], deterministic sampling [25], adversarial training [11], and others [6].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “[20] found that a GAN-like setup could improve samples from these models; Song et al.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “As noted by Jolicoeur-Martineau et al. (2020), FIDs can be signiﬁcantly worse without removing this noise.”
Denoising Diffusion Implicit Models (2020, methodology): “For the case of ˆ σ , the generated images seem to have more noisy perturbations under short trajectories; this explains why the FID scores are much worse than other methods, as FID is very sensitive to such perturbations (as discussed in Jolicoeur-Martineau et al. (2020)).”
Gotta Go Fast When Generating Data with Score-Based Models (2021, neutral): “Score-based generative models [Song and Ermon, 2019, 2020, Ho et al., 2020, Jolicoeur-Martineau et al., 2020, Song et al., 2020a, Piché-Taillefer, 2021] have been very successful at generating data from various modalities, such as images [Ho et al., 2020, Song et al., 2020a], audio [Chen et al.,…”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, methodology): “Jolicoeur-Martineau et al. (2021b) add an auxiliary adversarial loss to the main objective.”
Riemannian Score-Based Generative Modeling (2022, methodology): “The above showcases the basics of SGMs but we highlight that many improvements have been proposed; see (e.g. Song and Ermon, 2020; Jolicoeur-Martineau et al., 2021; Dhariwal and Nichol, 2021).”
Diffusion models for Handwriting Generation (2020, background): “(Jolicoeur-Martineau et al., 2020) combined the NCSN objective with an adversarial objective to address image generation.”
DiffusionCLIP: Text-guided Image Manipulation Using Diffusion Models (2021, background): “…(DDPM) (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based generative models (Song & Ermon, 2019; Song et al., 2020b) have shown great success in image generation (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2020a;b; Jolicoeur-Martineau et al., 2020; Dhariwal & Nichol, 2021).”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “Recent advances in training non-normalized probabilistic models, such as score-based generative models (Song & Ermon, 2019; 2020; Song et al., 2021; Ho et al., 2020; Song et al., 2020; Jolicoeur-Martineau et al., 2021) and energy-based models (Ackley et al.”
Universal Speech Enhancement with Score-based Diffusion (2022, methodology): “To sample, we follow noise-consistent Langevin dynamics (Jolicoeur-Martineau et al., 2021), which corresponds to the recursion over N uniformly discretized time steps t n ∈ [0 , 1] , where we set η and β with the help of a hyper-parameter (cid:15) ∈ [1 , ∞ ) (Serrà et al., 2021a).”
Diffusion Models in Vision: A Survey (2022, background): “[85] introduce an adversarial objective along with denoising score matching to train score-based models.”
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (2021, methodology): “…audio and image synthesis tasks; see e.g. Song and Ermon (2019); Cai et al. (2020); Chen et al. (2021a); Kong et al. (2021); Gao et al. (2020); Jolicoeur-Martineau et al. (2021b); Ho et al. (2020); Song and Ermon (2020); Song et al. (2020, 2021); Niu et al. (2020); Durkan and Song (2021);…”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, methodology): “†: taken from Jolicoeur-Martineau et al. (2021a).”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, methodology): “, 2017] combining the improvements from Song et al. [2021] and Dhariwal and Nichol [2021].”
Adversarial Diffusion Distillation (2023, methodology): “To improve quality, a discriminator loss is added to the score matching objective in Adversarial Score Matching [24] and the consistency objective of CTM [29].”