Non-Local Recurrent Network for Image Restoration

By Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, Thomas S. Huang. 2018

701 citations recorded by the source corpus.

Many classic methods have shown non-local self-similarity in natural images to be an effective prior for image restoration. However, it remains unclear and challenging to make use of this intrinsic property via deep networks. In this paper, we propose a non-local recurrent network (NLRN) as the first attempt to incorporate non-local operations into a recurrent neural network (RNN) for image restoration. The main contributions of this work are: (1) Unlike existing methods that measure self-similarity in an isolated manner, the proposed non-local module can be flexibly integrated into existing deep networks for end-to-end training to capture deep feature correlation between each location and its neighborhood. (2) We fully employ the RNN structure for its parameter efficiency and allow deep feature correlation to be propagated along adjacent recurrent states. This new design boosts robustness against inaccurate correlation estimation due to severely degraded images. (3) We show that it is essential to maintain a confined neighborhood for computing deep feature correlation given degraded images. This is in contrast to existing practice that deploys the whole image. Extensive experiments on both image denoising and super-resolution tasks are conducted. Thanks to the recurrent non-local operations and correlation propagation, the proposed NLRN achieves superior results to state-of-the-art methods with much fewer parameters.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, methodology): “Following several recent image restoration networks [18, 31, 14], we do not reduce the spatial resolution of the input.”
Relational recurrent neural networks (2018, background): “What is the Nth farthest from vector m? x = 339 for [19]: x += 597 for[94]: x += 875 x if 428 < 778 else 652 print(x)”
Efficient Attention: Attention with Linear Complexities (2018, background): “Subsequent works applied it to various fields in computer vision, including image restoration [16], video person re-identification [14], generative adversarial image modeling [28, 2], image de-raining [13], and few-shot learning [9, 11], etc.”
IR-NAS: Neural Architecture Search for Image Restoration (2019, methodology): “Following several recent image restoration network [7, 26, 15], we do not reduce the spatial resolution of the input.”
Lightweight Image Super-Resolution with Information Multi-distillation Network (2019, background): “Liu [19] explored the effectiveness of non-local module applied to image restoration.”
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020, background): “Following the recent success of transformer architecture [21] in natural language processing domain, it has been introduced in image processing tasks as well [15, 11].”
Neural Nearest Neighbors Networks (2018, background): “Recently, non-local neural networks have been proposed for higher-level vision tasks such as object detection or pose estimation [42] and, with a recurrent architecture, for low-level vision tasks [26].”
Efficient Long-Range Attention Network for Image Super-resolution (2022, methodology): “We set θ = ϕ and calculate the SA in the symmetric embedded Gaussian space [30, 5], which can save one 1 × 1 convolution in each In this way, we can avoid 2 n reshape and n 1 × 1 convolution operations for the following n SAs.”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “To address this problem, several recent works [31, 39, 83] start to employ single or few selfattention layers in low resolution feature maps due to the self-attention computational complexity being quadratic to the feature map size.”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, background): “Image Restoration typically includes super-resolution [14, 50, 62, 51, 75, 10, 23, 52], denoising [72, 44, 27], deblurring [67, 40, 60] and compression removal [13, 22].”
Residual Feature Aggregation Network for Image Super-Resolution (2020, methodology): “Actually, there are works [19, 3] that have attempted to use the Non-Local block to model pixel-wise similarities in image SR.”
CLEARER: Multi-Scale Neural Architecture Search for Image Restoration (2020, methodology): “As a matter of fact, the advances of image restoration in recent years are beneﬁted from the developments of various handcrafted neural network architectures [27, 14, 15, 19, 24, 40, 31, 34].”
Learning to Restore Low-Light Images via Decomposition-and-Enhancement (2020, methodology): “Deep learning has also been widely applied to the denoising problem [33, 49, 50, 27, 37, 32].”
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration (2023, methodology): “Apart from the traditional methods, nonlocal operations are also introduced into deep neural networks for video classification [70] and image SR [45, 85].”
Interpreting Super-Resolution Networks with Local Attribution Maps (2020, background): “Recently, nonlocal operations [54] are also introduced to SR networks [33, 60, 37], expecting to capture long-range information in the image and utilize self-similarity prior of natural images.”
Factorized Attention: Self-Attention with Linear Complexities (2018, neutral): “Subsequent works applied it to various fields in computer vision, including image restoration [20], de-raining [15], video person re-identification [17], salient segmentation [14], and cardiac motion scoring [35].”
Learning Texture Transformer Network for Image Super-Resolution (2020, background): “[18] proposed a non-local recurrent network for image restoration.”
EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, background): “Attention has proven its effectiveness in many tasks [43, 47, 22, 23, 51].”
Denoising Gravitational Waves with Enhanced Deep Recurrent Denoising Auto-encoders (2017, background): “Different from the conventional image and audio denoising setups [13, 14].”