MemNet: A Persistent Memory Network for Image Restoration

By Ying Tai, Jian Yang, Xiaoming Liu, Chunyan Xu. 2017

1,738 citations recorded by the source corpus.

Recently, very deep convolutional neural networks (CNNs) have been attracting considerable attention in image restoration. However, as the depth grows, the longterm dependency problem is rarely realized for these very deep models, which results in the prior states/layers having little influence on the subsequent ones. Motivated by the fact that human thoughts have persistency, we propose a very deep persistent memory network (MemNet) that introduces a memory block, consisting of a recursive unit and a gate unit, to explicitly mine persistent memory through an adaptive learning process. The recursive unit learns multi-level representations of the current state under different receptive fields. The representations and the outputs from the previous memory blocks are concatenated and sent to the gate unit, which adaptively controls how much of the previous states should be reserved, and decides how much of the current state should be stored. We apply MemNet to three image restoration tasks, i.e., image denosing, super-resolution and JPEG deblocking. Comprehensive experiments demonstrate the necessity of the MemNet and its unanimous superiority on all three tasks over the state of the arts. Code is available at https://github.com/tyshiwo/MemNet.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, methodology): “Motivated by this, most recent works on image denoising have shifted their approaches to deep learning, which builds a mapping function from noisy images to the desired corresponding clean images with deep learning models and have often outperformed conventional methods significantly [27, 33, 22].”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, background): “ssed a variety of network architectures, such as a deeper network with residual learning [5], Laplacian pyramid structure [6], residual blocks [1], recursive learning [7,8], densely connected network [9], deep back projection [10] and residual dense network [11]. Specically, Lim et al. [20] propose ”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, methodology): “LapSRN [6] MSLapSRN [7] ENet-PAT [8] MemNet [9] EDSR [10] SRMDNF [11] RCAN (ours)”
Residual Feature Aggregation Network for Image Super-Resolution (2020, methodology): “introduced recursive blocks in DRRN [24] and memory blocks in MemNet [25].”
Lightweight Image Super-Resolution with Information Multi-distillation Network (2019, methodology): “9749 MemNet [25] 678K 37.”
Fast and Accurate Single Image Super-Resolution via Information Distillation Network (2018, methodology): “, VDSR [12], DRCN [13], LapSRN [15], DRRN [22] and MemNet [23] as illustrated in Figure 1.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “[21], GANs have shown promising results in various areas [48], such as unconditional image synthesis [21, 23, 32, 33, 34, 49, 61], image translation [8, 12, 27, 29, 46, 47, 58, 67, 73, 84, 102, 103], text-to-image translation [62, 90, 95], image processing [17, 18, 28, 36, 39, 41, 42, 44, 72, 78, 89”
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal (2017, background): “The densely connected path has also been proved effective in many low-level vision fields[56, 46].”
SwinIR: Image Restoration Using Swin Transformer (2021, positive): “Since pioneering work SRCNN [18] (for image SR), DnCNN [90] (for image denoising) and ARCNN [17] (for JPEG compression artifact reduction), a flurry of CNN-based models have been proposed to improve model representation ability by using larger and deeper neural network architecture designs, such as ”
LatticeNet: Towards Lightweight Image Super-Resolution with Lattice Block (2020, methodology): “In this section, we conduct extensive experiments on four publicly available SR benchmark datasets and compare with 12 state-of-the-art lightweight SR models: SRCNN [8], FSRCNN [9], VDSR [18], DRCN [19], LapSRN [20], DRRN [30], MemNet [31], IDN* [17] 4 , CARN [2], BSRN [5], FALSR [6], and IMDN [16].”
Learning a Single Convolutional Super-Resolution Network for Multiple Degradations (2017, methodology): “Other CNN-based SISR methods can be found in [6, 22, 23, 30, 37, 42, 44, 45, 46, 53, 58].”
Deep Learning for Image Super-Resolution: A Survey (2019, methodology): “[77] feature similarity MS-SSIM [74] multi-scale SSIM IFC [75] information ﬁdelity criterion PSNR peak signal-to-noise ratio IQA image quality assessment SSIM [62] structural similarity MOS mean opinion score VIF [76] visual information ﬁdelity MSSIM [62] mean SSIM BN [140] batch normalization GAN [”
Edge-Enhanced GAN for Remote Sensing Image Superresolution (2019, background): “CNNs have been widely studied for feature extraction in recent years, such as the highway network [54], skip connection [55], dense-connection network [51], and memory network [56].”
Deep Learning on Image Denoising: An overview (2019, background): “(2017) [197] CNN Gaussian image denoisng, super-resolution and JPEG deblocking CNN with recursive unit, gate unit for image restoration Anwar et al.”
Non-Local Recurrent Network for Image Restoration (2018, methodology): “Training and testing protocols are followed as in [36].”
Denoising Prior Driven Deep Neural Network for Image Restoration (2018, methodology): “To exploit the long-term dependencies in the deep CNN, in [44] very deep persistent memory network containing memory blocks has been developed, leading to substantial improvements for typical”
Wide Activation for Efficient and Accurate Image Super-Resolution (2018, methodology): “…super-resolution, commonly only small image patches (e.g. 48 × 48 ) and small mini-batch size (e.g. 16) are used to speedup training [7, 14, 17, 19, 33, 36, 42], thus the mean and variance of small image patches differ a lot among mini-batches, making theses statistics unstable, which is…”