Score-based Generative Modeling in Latent Space

By Arash Vahdat, Karsten Kreis, Jan Kautz. 2021

888 citations recorded by the source corpus.

Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .

Subfields: Computer Vision and Pattern Recognition, Molecular Biology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “(iii) We show that, in contrast to previous work [87] which learns both an encoder/decoder architecture and a score-based prior simultaneously, our approach does not require a delicate weighting of reconstruction and generative abilities.”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[51] use a score-based model for the latent space of a VAE, while Rombach et al.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “A concurrent work [92] applied score-based generative modeling on the latent space of an VAE, and achieved similar improvements compared to regular VAEs.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “Such a behavior essentially roots in the different natures of the two metrics and has been investigated in extensive prior works (Theis et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021; Song et al., 2021; Vahdat et al., 2021; Watson et al., 2021; Kingma et al., 2021).”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “Improving the output quality and/or decreasing the computational cost of sampling are common topics in diffusion model research (e.g., [10, 24, 30, 31, 32, 36, 43, 50, 52, 53, 56]).”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “For instance, defining a low-dimensional latent variable for DPMs [34]; designing special diffusion processes with bounded score functions [35]; combining GANs with the reverse process of DPMs [36].”
Palette: Image-to-Image Diffusion Models (2021, background): “…models [Ho et al. 2020; Sohl-Dickstein et al. 2015; Song and Ermon 2020] have received a surge of recent interest [Austin et al. 2021; Cai et al. 2020; Hoogeboom et al. 2021; Kingma et al. 2021; Song et al. 2021; Vahdat et al. 2021], resulting in several key advances in modeling continuous data.”
Torsional Diffusion for Molecular Conformer Generation (2022, methodology): “Several methods [Salimans and Ho, 2022, Vahdat et al., 2021, Nichol and Dhariwal, 2021] have been proposed to improve and accelerate diffusion models in the domain of image generation.”
MagicVideo: Efficient Video Generation With Latent Diffusion Models (2022, result): “2021 [46] concurrently proposed a latent score-based model that shares similar ideas with LDM.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “To address this, cascaded [29] and latent [65, 88] diffusion models have been introduced.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, neutral): “There are other fast sampling methods by modifying the original DPMs to a latent space (Vahdat et al., 2021) or with momentum (Dockhorn et al., 2022).”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “Also, many other works have built EBMs or score-based models in the latent space of generative models, most of which, however, focus on improving generation quality with EBMs as a structural prior [2, 4, 19, 56, 55, 58, 53].”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, background): “In parallel, Denoising Diffusion Probabilistic Models (DDPMs) [2, 7, 35, 67, 72, 77] have demonstrated impressive generative capabilities, level of details, and diversity on top of GAN [26, 61, 62, 71, 73, 76].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Recently, research efforts on image synthesis have shifted from utilizing GANs [17], VAEs [29], and ﬂow models [13] to diffusion models [21, 50, 54, 73] due to more stable training, enhanced sample quality, and increased ﬂexibility in a conditional generation.”
Score Matching Model for Unbounded Data Score (2021, methodology): “UDM with high-dimensional CelebA-HQ 256 × 256 performs the state-of-the-art in FID out of the baseline models of the diffusion models (LSGM [21]) and the GAN models (PGGAN [30]).”
A Survey on Generative Diffusion Models (2022, methodology): “The joint objective is optimizing the ELBO or log-likelihood [35], [114].”
Regeneration Learning: A Learning Paradigm for Data Generation (2023, methodology): “• There are some methods that combine auto-encoders with diffusion models [75, 63], which first convert target data Y into latent vector Z (denoted as Z0) using VAE or VQ-VAE, and then gradually add noise on Z0 to get Zt.”
Diffusion Models for Adversarial Purification (2022, background): “These models have demonstrated strong sample quality, beating GANs in image generation (Dhariwal & Nichol, 2021; Vahdat et al., 2021).”
Infergrad: Improving Diffusion Models for Vocoder by Considering Inference in Training (2022, neutral): “Diffusion models have been extensively used for generation tasks [3, 17, 18, 19, 20, 21] since their capability of generating high-ﬁdelity samples matches the state-of-the-art autoregressive and GANs based methods.”