Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems

By Tim Meinhardt, Michael Möller, Caner Hazirbas, Daniel Cremers. IEEE International Conference on Computer Vision, 2017

394 citations recorded by the source corpus.

While variational methods have been among the most powerful tools for solving linear inverse problems in imaging, deep (convolutional) neural networks have recently taken the lead in many challenging benchmarks. A remaining drawback of deep learning approaches is their requirement for an expensive retraining whenever the specific problem, the noise level, noise type, or desired measure of fidelity changes. On the contrary, variational methods have a plug-and-play nature as they usually consist of separate data fidelity and regularization terms. In this paper we study the possibility of replacing the proximal operator of the regularization used in many convex energy minimization algorithms by a denoising neural network. The latter therefore serves as an implicit natural image prior, while the data term can still be chosen independently. Using a fixed denoising neural network in exemplary problems of image deconvolution with different blur kernels and image demosaicking, we obtain state-of-the-art reconstruction results. These indicate the high generalizability of our approach and a reduction of the need for problemspecific training. Additionally, we discuss novel results on the analysis of possible optimization algorithms to incorporate the network into, as well as the choices of algorithm parameters and their relation to the noise level the neural network is trained on.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Mathematical Physics

View Paper: https://www.semanticscholar.org/paper/2d582232e7a705769785fbc15c13b471f597b5f8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Mathematical Physics

Citations in the corpus

Deep Learning on Image Denoising: An overview (2019, methodology): “To achieve a tradeoff between efficiency and flexibility, a discriminative learning optimization-based method [156, 20] was presented for image applications, such as image denoising.”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “Such correspondence was originally applied in deep plugand-play (PNP) methods [44, 57, 74, 92, 94], which utilize trained denoiser to implicitly express the regularization term J(x) as a denoising problem.”
Denoising Prior Driven Deep Neural Network for Image Restoration (2018, methodology): “In [45], [46], a set ofDCNNmodels are pre-trained for imagedenoising task and are integrated into the optimization-based IR framework for different IR tasks.”
Model-Based Deep Learning (2020, background): “This can be implemented by incorporating deep denoisers as learned proximal mappings in iterative optimization, as carried out by plug-and-play networks1 [14], [15], [61]–[66].”
Learning to Optimize: A Primer and A Benchmark (2021, result): “Popular application examples include denoising [63; 172; 256; 51; 192; 126; 234; 134; 142], deblurring [63; 256; 146; 127; 58; 142; 162; 257; 121], super-resolution [175; 72; 172; 256; 51; 127; 257], inpainting [175; 172; 192; 3], and compressive sensing [175; 153; 43; 63; 171; 141; 255; 98; 142], J”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, methodology): “[98] replace the proximal operator of the regularization used in many convex energy minimization algorithms with a denoising neural network.”
Learned Primal-Dual Reconstruction (2017, methodology): “Examples of this include dictionary learning [9], but several alternative methods have been investigated, such as learning a variational auto-encoder [10] or using a cascade of”
Learning a Single Convolutional Super-Resolution Network for Multiple Degradations (2017, methodology): “An interesting line of CNN-based methods which can go beyond bicubic degradation adopt a CNN denoiser to solve SISR via model-based optimization framework [4, 34, 57].”
Deep Learning for Single Image Super-Resolution: A Brief Review (2018, methodology): “[95] with RMSE to conduct quantitative comparisons, and the representative qualitative comparisons are depicted in Fig.”
Denoising Diffusion Models for Plug-and-Play Image Restoration (2023, background): “In particular, the prior term can be implicitly addressed by Gaussian denoisers [39, 47, 54, 59].”
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers (2019, methodology): “Since then, other schemes such as the primal-dual method (Heide et al., 2014; Meinhardt et al., 2017; Ono, 2017), ADMM with increasing penalty parameter (Brif-man et al., 2016; Chan et al., 2017), generalized approximate message passing (Metzler et al., 2016), Newton iteration (Buzzard et al.,…”
One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models (2017, background): “[33] replace the proximal operator with a denoising network.”
Deep Mean-Shift Priors for Image Restoration (2017, methodology): “[33] that many algorithms that solve image restoration via MAP estimation only need the proximal operator of the regularization term, which can be interpreted as a MAP denoiser [22].”
Image-Adaptive GAN based Reconstruction (2019, neutral): “For example, a method which is not based on a generative model cannot produce a perceptually pleasing image of human face if the eyes are completely missing in an inpainting task (Yeh et al. 2017).”
Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels (2019, methodology): “During the past few years, a flurry of plug-and-play works have been developed from the following aspects: 1) different variable splitting algorithms, such as halfquadratic splitting (HQS) algorithm [1], alternating direction method of multipliers (ADMM) algorithm [8], FISTA [4], and primal-dual alg”
Deep Learning Techniques for Inverse Problems in Imaging (2020, methodology): “Improved methods for training deep neural networks for PnP can be found in [81], [106], [118].”