High-Resolution Image Synthesis with Latent Diffusion Models
By Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. Computer Vision and Pattern Recognition, 2021
26,251 citations recorded by the source corpus.
By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve new state of the art scores for image inpainting and class-conditional image synthesis and highly competitive performance on various tasks, including unconditional image generation, text-to-image synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models
View Paper: https://arxiv.org/abs/2112.10752
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Andreas Geiger
- Jonathan T. Barron
- Matthias Nießner
- Jun-Yan Zhu
- Eli Shechtman
- Alexei A. Efros
- Richard Zhang
- Daniel Cohen-Or
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Arash Vahdat
- Stefano Ermon
- Daniel Cohen-Or
- Alexei A. Efros
- Karsten Kreis
Citations in the corpus
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “They include but are not limited to generative models such as text-based language models (BLOOM [30], GPT-3 [40], OPT[271]), text-to-image models (ImaGen [205], DALL · E[170], Stable Diffusion [200]), and increasingly multimodal models [53] ( GPT-4 GPT-4 [ 171 ]).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “…to communicate with machines and may be of great relevance not only to LLMs but to other generalist models such as prompted image synthesizers (Rombach et al., 2022; Ramesh et al., 2022), for which public interest in prompt design and generation has also emerged (see Appendix A for examples).”
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Beyond NLP, transformers have also made big splashes as generative models in areas as diverse as text-to-image synthesis (Ramesh et al., 2022; Crowson et al., 2022; Rombach et al., 2022), protein modeling (Jumper et al.”
- Visual Instruction Tuning (2023, background): “To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [24, 14], with strong capabilities in open-world visual understanding such as classification [36, 18, 53, 50, 35], detection [26, 58, 29], segmentation [23, 59, 54] and captioning ”
- Scalable Diffusion Models with Transformers (2022, methodology): “Latent diffusion models (LDMs) [48] tackle this issue with a two-stage approach: (1) learn an autoencoder that compresses images into smaller spatial representations with a learned encoder E ; (2) train a diffusion model of representations z = E ( x ) instead of a diffusion model of images x ( E is…”
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “We examine CLIP-style models because of their importance in computer vision: CLIP-style models reach state-of-the-art performance on a wide range of image classification tasks [46, 65, 44, 7] and underlie image generation methods such as DALLE · 2 [49] and Stable Diffusion [51].”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “For example, the Pix2Pix model [35] is suitable for changing the style of an image.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “State-ofthe-art models in this space are either autoregressive [13, 50, 67, 72] or diffusion [43, 49, 51, 54, 79].”
- Generalized Decoding for Pixel, Image, and Language (2022, methodology): “It further empowers composite tasks like referring captioning using X-Decoder itself and image editing that combines with generative models such as Stable Diffusion [66] (labeled in yellow boxes).”
- Visual Programming: Compositional visual reasoning without training (2022, methodology): “Red modules use neural models (OWL-ViT [21], DSFD [18], MaskFormer [6], CLIP [23], ViLT [16], and Stable Diffusion [28]).”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Pair of images generated using StableDiffusion [52] with and without Prompt-to-Prompt [17].”
- LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Another recent success of multimodal learning is in image generation, where DALL-E [59] and later models [52, 60, 64, 66, 90] demonstrated the potential of text-guided image generation by producing high-quality images specific to the provided text.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Figure 3: Comparison between Imagen and DALL-E 2 [56], GLIDE [43], VQ-GAN+CLIP [12] and Latent Diffusion [59] on DrawBench: User preference rates (with 95% confidence intervals) for image-text alignment and image fidelity.”
- Foundation Models and Fair Use (2023, methodology): “Models such as GPT-3/4 (Brown et al., 2020; OpenAI, 2023), Stable Diffusion (Rombach et al., 2021), and Codex (Chen et al., 2021) are actively being integrated into a variety of products like Duolingo’s Language Learning App,3 Stability AI’s DreamStudio,4 GitHub’s CoPilot,5 and more.”
- The Gradient of Generative AI Release: Methods and Considerations (2023, methodology): “Model cards [58] have been popular as seen in Google’s PaLM [24], OpenAI’s GPT-2 [65] and GPT-3 [66], and Runway Research and Stability AI’s Stable Diffusion [81].”
- Scalable Adaptive Computation for Iterative Generation (2022, background): “Common diffusion models for images and videos can be roughly divided into pixel diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Dhariwal & Nichol, 2022; Ho et al., 2022a; Karras et al., 2022) and latent diffusion models (Rombach et al., 2022).”
- Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “Diffusion-based generative models [45] have emerged as a powerful new framework for neural image synthesis, in both unconditional [16, 36, 48] and conditional [17, 35, 36, 38, 39, 41, 42, 48] settings, even surpassing the quality of GANs [13] in certain situations [9].”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In CV, pre-trained models that learn from hundreds of millions of image-caption pairs have achieved top performance on image-to-text benchmarks [7–9], while delivering remarkably photo-realistic results in text-to-image tasks [10–14].”