Deblurring via Stochastic Refinement

By Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, A. Dimakis, Peyman Milanfar. 2021

384 citations recorded by the source corpus.

Image deblurring is an ill-posed problem with multiple plausible solutions for a given input image. However, most existing methods produce a deterministic estimate of the clean image and are trained to minimize pixel-level distortion. These metrics are known to be poorly correlated with human perception, and often lead to unrealistic reconstructions. We present an alternative framework for blind deblurring based on conditional diffusion models. Unlike existing techniques, we train a stochastic sampler that refines the output of a deterministic predictor and is capable of producing a diverse set of plausible reconstructions for a given input. This leads to a significant improvement in perceptual quality over existing state-of-the-art methods across multiple standard benchmarks. Our predict-and-refine approach also enables much more efficient sampling compared to typical diffusion models. Combined with a carefully tuned network architecture and inference procedure, our method is competitive in terms of distortion metrics such as PSNR. These results show clear benefits of our diffusion-based method for deblurring and challenge the widely used strategy of producing a single, deterministic reconstruction.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “[75] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar.”
Video Diffusion Models (2022, background): “1 INTRODUCTION Diffusion models have recently been producing high quality results in image generation and audio generation (e.g. Kingma et al., 2021; Saharia et al., 2021a;b; Dhariwal & Nichol, 2021; Ho et al., 2021; Nichol et al., 2021; Song et al., 2021b; Whang et al., 2021; Salimans & Ho, 2021; C”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “have broken the long-time dominance of generative adversarial networks (GANs) [73] in the challenging task of image synthesis [49, 90, 220, 225] and have also shown potential in a variety of domains, ranging from computer vision [2, 9, 17, 20, 91, 93, 115, 118, 137, 151, 161, 174, 200, 202, 245, 267”
MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model (2022, background): “However, recent studies show that it is also effective for the problems in which the ground-truth is unique, like superresolution [8] and deblurring [9].”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, neutral): “…including density estimation (Kingma et al., 2021), text-to-speech (Chen et al., 2021a; Kong et al., 2021; Chen et al., 2021b), image-to-image (Saharia et al., 2022c;a; Whang et al., 2022), and text-to-image (Rombach et al., 2022; Nichol et al., 2021; Ramesh et al., 2022; Saharia et al., 2022b).”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, background): “…2022, Nichol and Dhariwal, 2021, Ho et al., 2022, Dhariwal and Nichol, 2022, Ding et al., 2021], image-to-image generation [Saharia et al., 2022c,a, Whang et al., 2022], 3D synthesis [Poole et al., 2022, Watson et al., 2022], text-to-speech [Chen et al., 2021a, Kong et al., 2021, Chen et al.,…”
Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise (2022, methodology): “Recently, diffusion models have been applied to them [Song et al., 2021b] for the problems of deblurring, denoising, super-resolution, and compressive sensing [Whang et al., 2021, Kawar et al., 2021, Saharia et al., 2021, Kadkhodaie and Simoncelli, 2021].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Diffusion models can be directly applied to various inverse problems, such as superresolution [13, 64], inpainting [11, 47], deblurring [35, 82], and JPEG restoration [36, 62].”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “There has been one recent diffusion modeling work which studied size-agnostic blurred image restoration [33].”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “Another branch of training-required CDMs directly trains a new score estimator s ( x t , t, c ) conditioned on c [28, 16, 33, 34, 36, 45, 49, 20, 1, 13, 32, 29].”
Denoising Diffusion Restoration Models (2022, methodology): “More recently, denoising diffusion models were used to solve inverse problems in both supervised ( i.e. , degradation model is known during training) [42, 41, 13, 10, 54] and unsupervised settings [22, 26, 25, 21, 46, 47, 9].”