One Network to Solve Them All — Solving Linear Inverse Problems Using Deep Projection Models

By Jen-Hao Rick Chang, Chun-Liang Li, Barnab'as P'oczos, B. V. Vijaya Kumar, Aswin C. Sankaranarayanan. IEEE International Conference on Computer Vision, 2017

359 citations recorded by the source corpus.

While deep learning methods have achieved state-of-theart performance in many challenging inverse problems like image inpainting and super-resolution, they invariably involve problem-specific training of the networks. Under this approach, each inverse problem requires its own dedicated network. In scenarios where we need to solve a wide variety of problems, e.g., on a mobile camera, it is inefficient and expensive to use these problem-specific networks. On the other hand, traditional methods using analytic signal priors can be used to solve any linear inverse problem; this often comes with a performance that is worse than learning-based methods. In this work, we provide a middle ground between the two kinds of methods — we propose a general framework to train a single deep neural network that solves arbitrary linear inverse problems. We achieve this by training a network that acts as a quasi-projection operator for the set of natural images and show that any linear inverse problem involving natural images can be solved using iterative methods. We empirically show that the proposed framework demonstrates superior performance over traditional methods using wavelet sparsity prior while achieving performance comparable to specially-trained networks on tasks including compressive sensing and pixel-wise inpainting.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/a9d06af95a81a0e800162965878fe9a973235021

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

GAN-Based Projector for Faster Recovery With Convergence Guarantees in Linear Inverse Problems (2019, background): “This aspect is shared by two other neural-network-based solvers [35, 31], however, they model the image prior only implicitly by training a denoiser”
NETT: solving inverse problems with deep neural networks (2018, background): “Network cascades [32, 45] and trained iterative schemes [1, 2, 27, 12] learn free We initially submitted our paper a recognized journal in February 28, 2018.”
MoDL: Model-Based Deep Learning Architecture for Inverse Problems (2017, methodology): “The main difference of the proposed scheme with [15]–[17] is the use of end-to-end training, similar to [18] and [19].”
Denoising Diffusion Restoration Models (2022, methodology): “OneNet [39] is not included in the comparisons as it is limited to images of size 64× 64, and generalization to higher dimensions requires an improved network architecture.”
Solving ill-posed inverse problems using iterative deep neural networks (2017, background): “Some recent papers have approached learning the primal proximal operator [11, 42] in the scope of ADMM, but these do not consider learning the dual.”
Image-Adaptive GAN based Reconstruction (2019, positive): “the image prior in a way that does not require exhaustive ofﬂine training for each different observation model. This can be done by using CNN denoisers (Zhang et al. 2017; Meinhardt et al. 2017; Rick Chang et al. 2017) plugged into iterative optimization schemes (Venkatakrishnan, Bouman, and Wohlber”
Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods (2018, methodology): “’s [53] encoder-decoder CNN is trained in an adversarial learning context (discussed in the section “Using Generative Adversarial Networks to Learn Posteriors for the Inverse Problem”), it acquires a prior knowledge that is directly extracted from the statistics of the images seen in the training da”
Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction (2017, background): “More recently, several ideas were proposed, which integrate deep neural networks with model-based optimisation methods to solve inverse problems [20], [21].”
Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees (2018, background): “The recent papers [34, 35] resolve this issue by training a single quasi-projection operator to project each candidate solution on the manifold of natural images, and indeed in some sense is complementary to our approach.”
Learned D-AMP: Principled Neural Network based Compressive Image Recovery (2017, methodology): “The third method, which we denote OneNet [10], uses ADMM to solve several imaging inverse problems including inpainting, denoising, and CS.”
fastMRI: An Open Dataset and Benchmarks for Accelerated MRI (2018, background): “roved further by making use of this additional data. 7 Discussion MR image reconstruction is an inverse problem, and thus it has many connections to inverse problems in the computer vision literature [40,7,4,47], such as super-resolution, denoising and in-painting. In all of these inverse problems, ”
Deep Learning Techniques for Inverse Problems in Imaging (2020, background): “A similar approach is considered in [22], where a denoiser is learned from data via adversarial training.”
Deep Mean-Shift Priors for Image Restoration (2017, background): “Venkatakrishnan et al. [31] build on the ADMM algorithm and propose to replace the proximal operator of the regularizer with a denoiser such as BM3D or NLM. Unsurprisingly, this inspired several researchers to learn the proximal operator using CNNs [6, 22, 33, 38].”
Intermediate Layer Optimization for Inverse Problems using Deep Generative Models (2021, background): “Recent work has generalized this framework to solve more general inverse problems [1, 9, 10, 5, 2].”
ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing (2017, methodology): “It is worthwhile to note that ADMM-Net cannot be trivially extended to other CS domains, since it imposes a specific structure to the sampling matrix Φ. Utilizing the same training and 2https://github.com/yangyan92/Deep-ADMM-Net testing brain medical images as ADMM-Net, the CS-MRI results of ISTA-Ne”