Universal Guidance for Diffusion Models

By Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023

495 citations recorded by the source corpus.

Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available at github.com/arpitbansal297/Universal-Guided-Diffusion.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/d24b4f34197df0257390b57f02537e6ce3284f2e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “Various pre-trained models can be used to provide guidance signals, including classiﬁers, facial recognition models, and object detectors (Bansal et al., 2023).”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Diffusion-based image editing Diffusion models have also shown great potential for semantic editing [33, 3, 2, 31, 79, 37, 72, 68, 63, 42, 43, 5, 71].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Finally, building on top of diffusion, some recent works show controllable image generation from various other conditioning signals [3,24,57], such as depth or edge maps.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, methodology): “We compare FreeDoM with Universal Guidance Diffusion (UGD) [2] in style-guided generations.”
Training Diffusion Models with Reinforcement Learning (2023, unknown): “We use a clip range of 1e-4 for all experiments.”
DragDiffusion: Harnessing Diffusion Models for Interactive Point-Based Image Editing (2023, unknown): “In contrast to previous methods [3, 10, 34, 53], which apply gradient descent on latents of multiple diffusion steps, our approach focuses on optimizing the latent of one appropriately selected step to conveniently achieve the desired editing results.”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Note that, there are many concurrent works [23, 7, 3, 21] that studied a similar area.”
Practical and Asymptotically Exact Conditional Sampling in Diffusion Models (2023, unknown): “These approaches have had some success in inpainting problems (Meng et al., 2021; Song et al., 2020; Trippe et al., 2023), and other general inverse problems (Bansal et al., 2023; Chung et al., 2023; Ho et al., 2022; Song et al., 2023).”
Diffusion Self-Guidance for Controllable Image Generation (2023, methodology): “…the sampling process towards images that are more likely according to the classiﬁer [7], achieving a similar effect to truncation in GANs [3], and can also be applied with pretrained classiﬁers by ﬁrst denoising the intermediate noisy image (though this requires additional approximations [1]).”
CLeaR: A Unified Framework for Resolving the Leakage-Degradation Dilemma in Style Transfer (2026, unknown): “To address this issue, we introduce Energy-Guided Calibration (EGC), a test-time guidance mechanism inspired by prior diffusion guidance methods [48, 1].”