Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction

By Hyungjin Chung, Byeongsu Sim, Jong Chul Ye. Computer Vision and Pattern Recognition, 2021

468 citations recorded by the source corpus.

Diffusion models have recently attained significant interest within the community owing to their strong performance as generative models. Furthermore, its application to inverse problems have demonstrated state-of-the-art performance. Unfortunately, diffusion models have a critical downside - they are inherently slow to sample from, needing few thousand steps of iteration to generate images from pure Gaussian noise. In this work, we show that starting from Gaussian noise is unnecessary. Instead, starting from a single forward diffusion with better initialization significantly reduces the number of sampling steps in the reverse conditional diffusion. This phenomenon is formally explained by the contraction theory of the stochastic difference equations like our conditional diffusion strategy - the alternating applications of reverse diffusion followed by a non-expansive data consistency step. The new sampling strategy, dubbed Come-Closer-Diffuse-Faster (CCDF), also reveals a new insight on how the existing feed-forward neural network approaches for inverse problems can be synergistically combined with the diffusion models. Experimental results with super-resolution, image inpainting, and compressed sensing MRI demonstrate that our method can achieve state-of-the-art reconstruction performance at significantly reduced sampling steps.

Subfields: Mathematical Physics, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/96ba3947d48f5d6866556aa043ebbeaaf20e7759

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Physics

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

Citations in the corpus

Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “After surpassing GAN on image synthesis [50, 82, 200], diffusion model has shown great potential in numerous tasks, such as computer vision [14, 126, 252], natural language processing [9], temporal data modeling [32, 116, 174, 208], multi-modal modeling [10, 182, 265], robust learning [21, 107, 219]”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, background): “Recently, DM has become increasingly influential in the field of image restoration tasks, such as superresolution [28, 52] and inpainting [40, 50, 10].”
Diffusion Posterior Sampling for General Noisy Inverse Problems (2022, methodology): “Furthermore, while diffusion models were applied to various inverse problems including inpainting (Kadkhodaie & Simoncelli, 2021; Song et al., 2021b; Chung et al., 2022b; Kawar et al., 2022; Chung et al., 2022a), super-resolution (Kadkhodaie & Simoncelli, 2021; Choi et al., 2021; Chung et al.,…”
Diffusion Models in Vision: A Survey (2022, background): “To date, diffusion models have been applied to a wide variety of generative modeling tasks, such as image generation [1], [2], [3], [4], [5], [6], [7], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], image super-resolution [10], [23], [24], [25], [26], [27], image inpai”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, methodology): “Another popular way [7, 12, 13] is to use an unconditional pre-trained diffusion model as a prior and modify its reverse path to generate the expected HR image.”
Diffusion models in medical imaging: A comprehensive survey (2022, methodology): “• SDEs achieve better results in multi-modal conversion [49] • Cross-modality translation [50] • Conditional SDEs [50, 54, 58] • Solving linear inverse problems [52] • Using K-space data [178, 179] • Using manifold constraints [58] • Good initialization in the reverse process instead of random Gauss”
Score-based diffusion models for accelerated MRI (2021, background): “Interested readers are referred to Chung et al. (2021).”
A Survey on Generative Diffusion Models (2022, methodology): “For encoding degree optimization, approaches such as CCDF [78] and Franzese et al.”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “In image domains, various methods solve linear inverse problems by repeatedly injecting known information to the diffusion process [11, 12, 31, 32, 45, 78].”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “The diffusion-based SR meth-ods can be roughly summarized into two categories, concate-nating the LR image to the input of the denoiser [32, 33], and modifying the backward process of a pre-trained diffusion model [4, 5, 10, 14].”
Denoising Diffusion Restoration Models (2022, background): “, degradation model is known during training) [42, 41, 13, 10, 54] and unsupervised settings [22, 26, 25, 21, 46, 47, 9].”
Riemannian Score-Based Generative Modeling (2022, methodology): “In the Euclidean setting this idea has been successfully applied for several image processing problems such as deblurring, denoising or inpainting (see for instance Kawar et al., 2021a; Kawar et al., 2021b; Lee et al., 2021; Sinha et al., 2021; Batzolis et al., 2021; Chung et al., 2021).”
A Continuous Time Framework for Discrete Denoising Models (2022, background): “[17] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, methodology): “Similar to [9], our methods can also start the reverse diffusion process from a partial noised image rather than pure Gaussian noise to reduce the number of NFEs, especially for tasks like deblurring and SR.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “[27, 11, 14] attempt to use the cross-attention control to realize the conditional generation; [5, 6, 7, 8, 19, 24, 39, 41, 44, 43] directly modify the intermediate results to achieve zero-shot image restoration; [25] realizes image translation by adjusting the initial noisy images.”
Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, methodology): “Diffusion models have shown impressive performance both as generative models themselves [41, 13], and also as unsupervised inverse problem solvers [41, 8, 9, 25] that do not require problem-specific training.”