GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
By Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen. 2021
4,894 citations recorded by the source corpus.
Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at https://github.com/openai/glide-text2im.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Arash Vahdat
- Stefano Ermon
- Alexei A. Efros
- Daniel Cohen-Or
- Karsten Kreis
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “We note that applying classifier-free diffusion guidance [32] greatly boosts sample quality, such that the guided LDM-KL-8-G is on par with the recent state-of-the-art AR [26] and diffusion models [59] for text-to-image synthesis, while substantially reducing parameter count.”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Text-to-image generation [64, 72, 54, 65, 67, 45, 12, 41, 70] focuses on generating images from standalone text descriptions.”
- CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “We note that via this prompting, we can recover the full functionality of the DALL-E model proposed in Ramesh et al. (2021).”
- Scalable Diffusion Models with Transformers (2022, background): “Classifier-free guidance is widely-known to yield sig-nificantly improved samples over generic sampling techniques [21, 35, 46], and the trend holds for our DiT models.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “State-ofthe-art models in this space are either autoregressive [12, 46, 62, 67] or diffusion [39, 45, 47, 50, 74].”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Recent text-to-image diffusion models [41,48,51,54] have shown to generate realistic images from arbitrary text captions.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “A different line of research with considerable momentum involves diffusion-based text-to-image models, such as GLIDE [11] and concurrent works DALL-E 2 [12] (a.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Figure 3: Comparison between Imagen and DALL-E 2 [56], GLIDE [43], VQ-GAN+CLIP [12] and Latent Diffusion [59] on DrawBench: User preference rates (with 95% confidence intervals) for image-text alignment and image fidelity.”
- Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “Diffusion-based generative models [45] have emerged as a powerful new framework for neural image synthesis, in both unconditional [16, 36, 48] and conditional [17, 35, 36, 38, 39, 41, 42, 48] settings, even surpassing the quality of GANs [13] in certain situations [9].”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Moreover, we adapt the classifier-free guidance technique [53, 54].”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “Recent advances in large-scale generative models [6, 42, 50] have led to a major paradigm shift towards building general-purpose models, which can perform many new tasks not explicitly trained on.”
- AudioGen: Textually Guided Audio Generation (2022, methodology): “More recently, diffusion model have gained increased popularity (Nichol et al., 2022; Ramesh et al., 2022; Saharia et al.; Rombach et al., 2022).”
- Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “Diffusion generative models were first proposed in [11] and achieved strong results on image generation [13], [31], [32], [33] and speech synthesis [34], [35], [36], [37].”
- MusicLM: Generating Music From Text (2023, background): “The closest to our approach among these works is DALL · E 2 (Ramesh et al., 2022).”
- InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Diffusion models have been demonstrated as a strong generative model for image generation [33], [34], [35], [36], text generation [37], [38] and audio generation [23], [39], [40], [41].”
- Video Diffusion Models (2022, background): “Similar findings have been reported on text-to-image generation [36].”