Denoising Diffusion Restoration Models

By Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song. Neural Information Processing Systems, 2022

1,332 citations recorded by the source corpus.

Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5x faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set.

Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/3d3c5fcbc40aadccceda58d3d9c5cd00588ea0b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, background): “…synthetic image and audio generation (Dhariwal & Nichol, 2021; Saharia et al., 2022; Rombach et al., 2022), as well as in applications like media editing, restoration (Meng et al., 2021; Cheuk et al., 2023; Kawar et al., 2022; Saito et al., 2023; Hernandez-Olivan et al., 2023; Murata et al., 2023).”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “The diffusion-based SR meth-ods can be roughly summarized into two categories, concate-nating the LR image to the input of the denoiser [32, 33], and modifying the backward process of a pre-trained diffusion model [4, 5, 10, 14].”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, methodology): “As an early attempt, Kawar et al. [31] explore pre-trained DDPMs with variational inference, and achieve satisfactory results on multiple restoration tasks, but their Denoising Diffusion Restoration Model (DDRM) leverages the singular value decomposition (SVD) on a known linear degradation matrix, m”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, background): “Recently, DM has become increasingly influential in the field of image restoration tasks, such as superresolution [28, 52] and inpainting [40, 50, 10].”
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion (2023, positive): “For instance, DDRM [19] performs diffusion sampling in the spectral space of degradation operator A to reconstruct the missing information in the observation y .”
DeepCache: Accelerating Diffusion Models for Free (2023, background): “These models have demonstrated remarkable efficacy across diverse applications, being employed for the generation of images [21, 62, 66], text [11, 30], audio [6, 46], and video [19, 38, 58].”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, methodology): “Another popular way is to utilize an unconditional pre-trained diffusion model as a prior and incorporate additional constraint to guide the reverse process [7, 12, 13, 45].”
LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023, background): “Diffusion models [3, 10, 25, 30, 34– 36, 44] are being recognized as a promising family of generative models that have proven to be state-of-the-art sample quality for a variety of image generation benchmarks [6,41,45], including class-conditional image generation [7,48], text-to-image generation [2”
Universal Guidance for Diffusion Models (2023, background): “Prior work that studied guided image generation did so with a variety of restrictions and external guidance functions [3, 4, 6, 7, 12, 16, 28].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, methodology): “They have also been successfully used for various image editing and processing tasks [18,25,41,43,48,53,66,67,69,74,84].”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “In image domains, various methods solve linear inverse problems by repeatedly injecting known information to the diffusion process [11, 12, 31, 32, 45, 78].”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, positive): “Specifically, DDRM [35] samples the posterior distribution with a linear degradation operator.”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “2021) to solve the Real-ISR problems and led to interesting results (Chen et al. 2022; Kawar et al. 2022; Wang et al. 2023).”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, background): “…traction, as they advanced the state-of-the-art in image generation [16, 31, 61, 65], and have been deployed in various downstream applications such as image restoration [30, 52], adversarial pu-riﬁcation [11, 40], image compression [63], image classiﬁ-cation [69], and others [13,18,32,44,55,67].”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, methodology): “The data term can be solved independently, allowing DiffPIR to handle a wide range of degradation models with various degradation operators H .”