Non-local Color Image Denoising with Convolutional Neural Networks

By Stamatios Lefkimmiatis. 2016

356 citations recorded by the source corpus.

We propose a novel deep network architecture for grayscale and color image denoising that is based on a non-local image model. Our motivation for the overall design of the proposed network stems from variational methods that exploit the inherent non-local self-similarity property of natural images. We build on this concept and introduce deep networks that perform non-local processing and at the same time they significantly benefit from discriminative learning. Experiments on the Berkeley segmentation dataset, comparing several state-of-the-art methods, show that the proposed non-local models achieve the best reported denoising performance both for grayscale and color images for all the tested noise levels. It is also worth noting that this increase in performance comes at no extra cost on the capacity of the network compared to existing alternative deep network architectures. In addition, we highlight a direct link of the proposed non-local models to convolutional neural networks. This connection is of significant importance since it allows our models to take full advantage of the latest advances on GPU computing in deep learning and makes them amenable to efficient implementations through their inherent parallelism.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

Non-local Neural Networks (2017, methodology): “Block matching was used with neural networks for image denoising [6, 31].”
Deep Image Prior (2017, background): “The work [20] investigates the model that combines ConvNet with a self-similarity based denoising and thus also bridges the two groups of methods, but still requires training on a hold-out set.”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, background): “Image Restoration typically includes super-resolution [14, 50, 62, 51, 75, 10, 23, 52], denoising [72, 44, 27], deblurring [67, 40, 60] and compression removal [13, 22].”
A survey on instance segmentation: state of the art (2020, methodology): “This idea was successively developed by Block-matching 3D (BM3D) [102-105].”
Face De-Spoofing: Anti-Spoofing via Noise Modeling (2018, background): “General de-noising works assume additive Gaussian noise and researchers propose non-local ﬁlters [33] or CNNs [13,34] to exploit the inherent similarity within the images.”
Universal Denoising Networks : A Novel CNN Architecture for Image Denoising (2017, methodology): “We emphasize that as opposed to previous networks [6, 23, 38] that followed a similar unrolling strategy, our network architecture is based on a constrained minimization formulation (5) rather than on an unconstrained one (4).”
Real Image Denoising With Feature Attention (2019, background): “Currently, due to the popularity of convolutional neural networks (CNNs), image denoising algorithms [63, 64, 39, 14, 53, 8] have achieved a performance boost.”
Neural Nearest Neighbors Networks (2018, methodology): “These include the classical BM3D [9], which uses a hand-crafted denoising pipeline, and the state-of-the-art trainable non-local models NLNet [23] and UNLNet [24], both learning to process non-locally aggregated patches.”
Data Dropout: Optimizing Training Data for Convolutional Neural Networks (2018, background): “Typical works include MLP [18], CSF[19], NLNet [20], and DnCNN [21], which presented very competitive results.”
Deep Unfolding Network for Image Super-Resolution (2020, background): “…e.g ., half-quadratic splitting [2], alternating direction method of multipliers [6] and primal-dual [1, 9]) have been proposed to solve different image restoration tasks, such as image de-noising [11, 32], image deblurring [29, 49], image compressive sensing [61, 63], and image demosaicking [28].”
Meta-Transfer Learning for Zero-Shot Super-Resolution (2020, background): “Additionally, the powerful image prior of non-local property is embedded into network architecture [19, 22, 46] by implicitly learning such priors to boost the performance of the networks further.”
Deep Learning on Image Denoising: An overview (2019, background): “rning (RL) [68] was proposed to deal with image denoising, super-resolution, and JPEG image deblocking. Taking into account between denoising performance and speed, a color non-local network (CNLNet) [105] combined non-local self-similarity (NLSS) and CNN to efﬁciently remove color-image noise. In t”
NAS-DIP: Learning Deep Image Prior with Neural Architecture Search (2020, background): “Among the design choices, 2D transposed convolution is one of the popular choices for various dense prediction tasks, including semantic segmentation [54], optical flow estimation [21], depth prediction [15], and image restoration [49, 73] and synthesis [59] problems.”
Spatial-Adaptive Network for Single Image Denoising (2020, unknown): “s, learningbased denoising methods have developed rapidly. Some works combine natural priors with deep neural networks. TRND [7] introduced the ﬁeld-of-experts prior into a deep neural network. NLNet [16] combined the non-local selfsimilarity prior with a CNN. Limited by the designed priors, their p”
Non-Local Recurrent Network for Image Restoration (2018, methodology): “Image Denoising: For a fair comparison with other methods based on deep networks, we train our model under two settings: (1) We use the training data as in TNRD [7], DnCNN [49] and NLNet [23], and the result is shown in Table 4.”
Relation-Aware Global Attention for Person Re-Identification (2019, background): “Exploration of non-local/global information has been demonstrated to be very useful for image denoising [3, 8, 4], texture synthesis [10], super-resolution [14], inpainting [2], and even high level tasks such as image recognition, object segmentation [35] and action localization [7].”
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising (2017, unknown): “Recently, Lefkimmiatis [26] and Qiao et al.”