Classifier-Free Diffusion Guidance
By Jonathan Ho. 2022
6,753 citations recorded by the source corpus.
Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of a diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by a pure generative model without such a classifier: in what we call classifier-free guidance, we jointly train a conditional and an unconditional diffusion model, and we combine the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity similar to that obtained using classifier guidance.
Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Statistical and Nonlinear Physics
- Arash Vahdat
- Karsten Kreis
- Stefano Ermon
- Jonathan Ho
- Yang Song
- Sergey Levine
- David J. Fleet
- Prafulla Dhariwal
In Generative Models
- Arash Vahdat
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Stefano Ermon
- Karsten Kreis
- Alexei A. Efros
- Daniel Cohen-Or
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “We note that applying classifier-free diffusion guidance [31] greatly boosts sample quality.”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “We enable classifier-free guidance [24] by randomly setting the CLIP embeddings to zero (or a learned embedding) 10% of the time, and randomly dropping the text caption 50% of the time.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “…image filtering (utilizing CLIP [44] for instance), we extend the use of classifier-free guidance suggested for diffusion models [53, 20] by [22, 41] to transformers, eliminating the need for post-generation filtering, thus producing faster and higher quality generation results, better…”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “Ho & Salimans (2021) achieved similar results without a separately trained classifier through the use of classifier-free guidance , a form of guidance that interpolates between predictions from a diffusion model with and without labels.”
- Scalable Diffusion Models with Transformers (2022, methodology): “In this setting, classifier-free guidance can be used to encourage the sampling procedure to find x such that log p(c|x) is high [21].”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Classifier-free diffusion guidance [20] is a method for trading off the quality and diversity of samples generated by a diffusion model.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Classifier-free guidance [37] (CF-guidance in short) is critical in the context of improving the sample quality of diffusion models [11, 12, 13] without pretrained classifiers.”
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “All diffusion models are conditioned on the text embedding sequence and use classifier-free guidance [27].”
- Scalable Adaptive Computation for Iterative Generation (2022, methodology): “It is known to be challenging to train pixel diffusion models for high resolution images on ImageNet without guidance (Dhariwal & Nichol, 2022; Ho & Salimans, 2021) or cascades (Ho et al., 2022a).”
- Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “…sampler improvements alone, we are able to bring the ImageNet-64 model that originally achieved FID 2.07 [9] to 1.55 that is very close to the state-of-the-art; previously, FID 1.48 has been reported for cascaded diffusion [17], 1.55 for classifier-free guidance [18], and 1.52 for StyleGAN-XL [44].”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Moreover, we adapt the classifier-free guidance technique [53, 54].”
- AudioGen: Textually Guided Audio Generation (2022, methodology): “As pointed out by Ho & Salimans (2021), the choice of the CFG scale offers a trade-off between sample diversity and quality with respect to the conditioning text.”
- InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “To solve this problem, we introduce the classifier free guidance [58], [59] to enhance the connection between x 0 and y .”
- Video Diffusion Models (2022, methodology): “The additional gradient term in this expression can be interpreted as a form of guidance [16, 20] based on the model’s reconstruction of the conditioning data, and we therefore refer to this method as reconstruction-guided sampling, or simply reconstruction guidance.”
- Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “For classifier-free guidance shown in (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), by jointly training a conditional and an unconditional diffusion model, it could be possible to combine the conditional and unconditional scores to attain a trade-off between sample quality and diversity.”
- Improved Distribution Matching Distillation for Fast Image Synthesis (2024, background): “Moreover, pure GAN-based methods also lack a natural way to incorporate classifier-free guidance [72], essential for high-quality text-to-image synthesis [1,2].”