Improving Diffusion Models for Inverse Problems using Manifold Constraints

By Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, Jong Chul Ye. Neural Information Processing Systems, 2022

698 citations recorded by the source corpus.

Recently, diffusion models have been used to solve various inverse problems in an unsupervised manner with appropriate modifications to the sampling process. However, the current solvers, which recursively apply a reverse diffusion step followed by a projection-based measurement consistency step, often produce suboptimal results. By studying the generative sampling path, here we show that current solvers throw the sample path off the data manifold, and hence the error accumulates. To address this, we propose an additional correction term inspired by the manifold constraint, which can be used synergistically with the previous solvers to make the iterations close to the manifold. The proposed manifold constraint is straightforward to implement within a few lines of code, yet boosts the performance by a surprisingly large margin. With extensive experiments, we show that our method is superior to the previous methods both theoretically and empirically, producing promising results in many applications such as image inpainting, colorization, and sparse-view computed tomography. Code available https://github.com/HJ-harry/MCG_diffusion

Subfields: Computational Theory and Mathematics, Mathematical Physics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/b3f5cf32178bcbed91aa5303b70963c6463f48a2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Mathematical Physics

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Diffusion models can be directly applied to various inverse problems, such as superresolution [13, 64], inpainting [11, 47], deblurring [35, 82], and JPEG restoration [36, 62].”
Diffusion models in medical imaging: A comprehensive survey (2022, methodology): “• SDEs achieve better results in multimodal conversion [49] • Cross-modality translation [50] • Conditional SDEs [50, 54, 58] • Solving linear inverse problems [52] • Using K-space data [147, 148] • Using manifold constraints [58] • Good initialization in the reverse process instead of random Gaussi”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “In this study, we present StableSR , an approach that preserves pre-trained diffusion priors without making explicit assumptions about the degradations.”
Universal Guidance for Diffusion Models (2023, background): “Prior work that studied guided image generation did so with a variety of restrictions and external guidance functions [3, 4, 6, 7, 12, 16, 28].”
A Survey on Generative Diffusion Models (2022, methodology): “They have been successfully used in super-resolution [58], [160], classiﬁcation [162], and noise robustness [161], [163].”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “[27, 11, 14] attempt to use the cross-attention control to realize the conditional generation; [5, 6, 7, 8, 19, 24, 39, 41, 44, 43] directly modify the intermediate results to achieve zero-shot image restoration; [25] realizes image translation by adjusting the initial noisy images.”
Diffusion Posterior Sampling for General Noisy Inverse Problems (2022, methodology): “…2022a), compressed-sensing MRI (CS-MRI) (Song et al., 2022; Chung & Ye, 2022; Chung et al., 2022b), computed tomography (CT) (Song et al., 2022; Chung et al., 2022a), etc., to our best knowledge, all works so far considered linear inverse problems only, and have not explored nonlinear inverse…”
DiffVoice: Text-to-Speech with Latent Diffusion (2023, methodology): “As duration is jointly modeled with other factors of speech, generic inverse problem solving algorithms with DMs [14, 15] can be directly combined with DiffVoice for solving inverse problems in speech synthesis, including text-based speech editing, and zero-shot adaptation.”