ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

By Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon. 2021

969 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPM) have shown remarkable performance in unconditional image generation. However, due to the stochasticity of the generative process in DDPM, it is challenging to generate images with the desired semantics. In this work, we propose Iterative Latent Variable Refinement (ILVR), a method to guide the generative process in DDPM to generate high-quality images based on a given reference image. Here, the refinement of the generative process in DDPM enables a single DDPM to sample images from various sets directed by the reference image. The proposed ILVR method generates high-quality images while controlling the generation. The controllability of our method allows adaptation of a single DDPM without any additional learning in various image generation tasks, such as generation from various downsampling factors, multi-domain image translation, paint-to-image, and editing with scribbles.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction (2021, methodology): “We also perform a comparison study where we set the total number of diffusion steps to N = 1000 starting from T = 1 for ILVR [5], and set t0 to 0.”
Diffusion Models for Medical Anomaly Detection (2022, methodology): “In the flow of this success, they were also applied on image-to-image translation [23,7], segmentation [4], reconstruction [22] and registration[12].”
Diffusion models as plug-and-play priors (2022, background): “In [5], generation is conditioned on an auxiliary image by guiding the denoising process through correction steps that match the low-frequency components of the generated and conditioning images.”
Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, background): “Additionally, DDPMs can also be used to generate conditional samples, too (Jeong et al., 2021; Choi et al., 2021).”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, methodology): “Similar to EGSDE [50] and ILVR [5], we choose a low-pass filter K ( · ) in this setup.”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “Avrahami et al., 2022c; Choi et al., 2021; Mokady et al., 2022; Couairon et al., 2022; Kong et al., 2023; Kwon et al., 2022) or by performing lightweight model finetuning (Ruiz et al., 2022; Kawar et al., 2022; Kim et al., 2022; Valevski et al., 2022).”
Denoising Diffusion Restoration Models (2022, methodology): “ILVR [9] suggests a diffusion-based method that handles noiseless super-resolution, and can run in 250 steps.”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, positive): “In this study, we extend our exploration beyond the realm of content creation and examine the potential beneﬁts of using the diffusion prior for super-resolution (SR).”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, methodology): “Zero-Shot Editing Some methods [33, 8, 4, 31, 14] use diffusion-based generate models to achieve zero-shot editing.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, positive): “By applying iterative latent variable sampling [11] to”
Pix2Video: Video Editing using Image Diffusion (2023, methodology): “In another line of work, methods have focused on editing images while preserving structures via attention layer manipulation [16, 51], additional guidance optimization [7], or per-instance finetuning [25].”
Zero-shot Image-to-Image Translation (2023, methodology): “Other methods [Choi et al. 2021; Song et al. 2020] proposes to modify the diffusion process by incorporating conditioning user inputs but have been only applied to single-category models.”
A Survey on Generative Diffusion Models (2022, background): “It is commonly used in denoising [58], paint-to-image [59], and inpainting tasks [17].”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “The diffusion-based SR meth-ods can be roughly summarized into two categories, concate-nating the LR image to the input of the denoiser [32, 33], and modifying the backward process of a pre-trained diffusion model [4, 5, 10, 14].”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, neutral): “synthesizing data in user-provided masks, while keeping the rest of the image intact [6, 14, 39].”