More Control for Free! Image Synthesis with Semantic Diffusion Guidance

By Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, Trevor Darrell. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2021

339 citations recorded by the source corpus.

Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores, without re-training the diffusion model. We explore CLIP-based language guidance as well as both content and style-based image guidance in a unified framework. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis, synthesis of images related to a style or content reference image, and examples with both textual and image guidance.1

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dc563ae2179a79a3b58749bc142a719e009d6cc5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Diffusion Models already have a Semantic Latent Space (2022, background): “Classiﬁer guidance manipulates images by imposing gradients of a classiﬁer on the latent variables in the reverse process to match the target class (Dhariwal & Nichol, 2021; Avrahami et al., 2022; Liu et al., 2021).”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “This formulation is also similar to classifier-free conditional generation [25], in which λ plays the role of a gradient weight to control the intensity of the conditioning.”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, background): “[37] guide a diffusion process with a text and an image, synthesising images similar to the given one, and aligned with the given text.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “More following work are proposed using transformers [10, 11] or diffusion models [18, 3, 5, 17, 19, 23, 25] to generate high-fidelity images from text.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, methodology): “Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “Similarly, in the T2I subdomain, Liu et al. (2021b) and Nichol et al. (2021) apply a noisy CLIP model to measure the cosine similarity between text prompts and noisy images.”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, methodology): “fusion models is the commonly used one [15,46,84], where the guidance is conditioned on xt but with the absence of Σ, named GDP-xt.”
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, background): “Recently powerful controlling mechanisms [47, 22, 17] emerged to guide the diffusion process for text-to-image generation.”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “ance, some works [15], [61], [62] have also explored crossmodal guidance with CLIP [45].”
Blended Diffusion for Text-driven Editing of Natural Images (2021, methodology): “Concurrently with our work, Liu et al. [32] and Kim et al. [29] propose ways to utilize a diffusion model in order to perform global text-guided image manipulations.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “[39] propose a diffusion-based technique allowing for image variations guided by reference image or text.”