Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels

By Kai Zhang, Wangmeng Zuo, Lei Zhang. 2019

400 citations recorded by the source corpus.

While deep neural networks (DNN) based single image super-resolution (SISR) methods are rapidly gaining popularity, they are mainly designed for the widely-used bicubic degradation, and there still remains the fundamental challenge for them to super-resolve low-resolution (LR) image with arbitrary blur kernels. In the meanwhile, plug-and-play image restoration has been recognized with high flexibility due to its modular structure for easy plug-in of denoiser priors. In this paper, we propose a principled formulation and framework by extending bicubic degradation based deep SISR with the help of plug-and-play framework to handle LR images with arbitrary blur kernels. Specifically, we design a new SISR degradation model so as to take advantage of existing blind deblurring methods for blur kernel estimation. To optimize the new degradation induced energy function, we then derive a plug-and-play algorithm via variable splitting technique, which allows us to plug any super-resolver prior rather than the denoiser prior as a modular part. Quantitative and qualitative evaluations on synthetic and real LR images demonstrate that the proposed deep plug-and-play super-resolution framework is flexible and effective to deal with blurry LR images.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “As mentioned in the introduction, bicubic downsampling and traditional degradations underly existing DNNsbased SISR methods [22, 34, 41, 55], or some simple variants do [10, 11, 38, 47, 53, 54].”
Unfolding the Alternating Optimization for Blind Super Resolution (2020, methodology): “To make this problem easier to be solved, previous methods [37, 32, 38, 35] usually decompose it into two sequential steps: i) estimating blur kernel from LR image and ii) restoring SR image based on estimated kernel.”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, background): “In particular, the prior term can be implicitly addressed by Gaussian denoisers [39, 47, 54, 59].”
Learning Filter Basis for Convolutional Neural Network Compression (2019, background): “Recently, deep convolutional neural network (CNN) based approaches have been setting new state-of-the-art results not only for high-level computer vision tasks such as image classification [25, 44, 15, 18], segmentation [11, 35], and object detection [11, 10, 43, 42], but also for low-level tasks su”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “Such correspondence was originally applied in deep plugand-play (PNP) methods [44, 57, 74, 92, 94], which utilize trained denoiser to implicitly express the regularization term J(x) as a denoising problem.”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, background): “Various SR technologies were explored from different perspectives, including network architecture [31, 32, 33, 34], image prior [35, 36, 37, 38], deep unfolding [39, 40, 41], degradation model [17, 18, 42, 43].”
Learned Image Downscaling for Upscaling Using Content Adaptive Resampler (2019, background): “, the SRCNN, many state-ofthe-art neural SR models have been proposed [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
Learning to Optimize: A Primer and A Benchmark (2021, neutral): “Popular application examples include denoising [63; 172; 256; 51; 192; 126; 234; 134; 142], deblurring [63; 256; 146; 127; 58; 142; 162; 257; 121], super-resolution [175; 72; 172; 256; 51; 127; 257], inpainting [175; 172; 192; 3], and compressive sensing [175; 153; 43; 63; 171; 141; 255; 98; 142], J”
Deep Unfolding Network for Image Super-Resolution (2020, methodology): “Perhaps the most ﬂexible CNN-based works which can handle various blur kernels, scale factors and noise levels, are the deep plug-and-play methods [65, 68].”
Blind Face Restoration via Deep Multi-scale Component Dictionaries (2020, background): “, image super-resolution [9,19,24,44, 46], denoising [13, 38, 42, 43], deblurring [22, 29, 41], and compression artifact removal [8, 10, 12].”
Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study (2019, background): “The tjﬁrst team (ranked No. 12, mAP 26.50) referred to a more sophisticated enhancement module (ﬁrst enhancing illumination by LIME [107], then super-resolving by DPSR [162], ended by denoising with BM3D [4]), followed by aggregating the DSFD-detection results on the original and enhanced images.”
Deep Learning on Image Denoising: An overview (2019, methodology): “lution image. To enhance the raw sensor data, Kokkinos et al. [97] presented residual CNN with iterative algorithm for image demosaicking and denoising. To handle arbitrary blur kernels, Zhang et al. [226] proposed to use cascaded deblurring and SISR networks to recover plug-and-play super-resolutio”
Real-World Super-Resolution via Kernel Estimation and Noise Injection (2020, background): “To overcome these challenges of real-world super-resolution, recent work [42, 49] combined with denoising or deblurring have been proposed.”