Neural Nearest Neighbors Networks

By Tobias Plötz, Stefan Roth. 2018

376 citations recorded by the source corpus.

Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w.r.t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w.r.t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, positive): “Following several recent image restoration networks [18, 31, 14], we do not reduce the spatial resolution of the input.”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, neutral): “All elements of the knowledge source E are pre-computed and kept ﬁxed — we do not backpropagate to affect the embeddings of the pre-encoded knowledge.”
IR-NAS: Neural Architecture Search for Image Restoration (2019, methodology): “Following several recent image restoration network [7, 26, 15], we do not reduce the spatial resolution of the input.”
Generalizable Adversarial Attacks with Latent Variable Perturbation Modelling (2019, methodology): “To address this issue, we develop a differentiable nearest neighbour attention mechanism (inspired by [ 22 ] ), where for each word x i in the input sequence x , we deﬁne the combination function c ( x i , δ i ) with a tunable temperature parameter τ as: where α w denotes the angular distance…”
Residual Dense Network for Image Restoration (2018, methodology): “Moreover, MWCNN [44], N(3)Net [45], and NLRN [46] also have achieved good performance for some noise levels, which indicates that Wavelet transformation and non-local processing are promissing candidates for further image denoising improvements.”
Learning to Restore Low-Light Images via Decomposition-and-Enhancement (2020, methodology): “Deep learning has also been widely applied to the denoising problem [33, 49, 50, 27, 37, 32].”
Spatial-Adaptive Network for Single Image Denoising (2020, positive): “Therefore, some recent works have made progress in real image denoising [26,39,12,4].”
Unprocessing Images for Learned Raw Denoising (2018, methodology): “We see that our models produce a 14% and 25% reduction in error on the two raw metrics compared to the next best performing technique (N3Net [32]), and a 21% and 38% reduction in error on the two sRGB metrics compared”
Plug-and-Play Image Restoration With Deep Denoiser Prior (2020, methodology): “In [57], [58], [59], [60], the non-local module was incorporated into the network design for better restoration.”
Real Image Denoising With Feature Attention (2019, background): “Recently, many algorithms focused on blind denoising on real-noisy images [50, 31, 12].”
Variational Denoising Network: Toward Blind Noise Modeling and Removal (2019, methodology): “It should be noted that our proposed method is a general framework, most of the commonly used network architectures [41, 31, 24, 42] in image restoration can also be easily substituted.”
Learning Enriched Features for Fast Image Restoration and Enhancement (2022, neutral): “Recently, deep learning models [6], [9], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42] make significant advances in image denoising, yielding”
FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation (2019, background): “More recently, the approaches proposed in [35, 29] combine neural architectures with non-local techniques.”
CLEARER: Multi-Scale Neural Architecture Search for Image Restoration (2020, background): “In details, BM3D [4], WNNM [8], RED [27], MemNet [40], NLRN [19], DuRN-P [24], N3Net [31], HiNAS [46], and E-CAE [38].”
SwinIR: Image Restoration Using Swin Transformer (2021, methodology): “Compared methods include traditional models BM3D [14] and WNNM [29], CNN-based models DnCNN [90], IRCNN [91], FFDNet [92], N3Net [65], NLRN [52], FOCNet [38], RNAN [96], MWCNN [54] and DRUNet [88].”
CycleISP: Real Image Restoration via Improved Data Synthesis (2020, methodology): “For instance, the typical procedure to acquire noisy paired data is to take multiple noisy images (a) Noisy Input (b) N3NET [45] PSNR(RAW) / PSNR(sRGB) 38.24 dB / 32.42 dB (c) UPI [7] (d) Ours 37.37 dB / 35.49 dB 40.44 dB / 36.16 dB Figure 1: Denoising a real camera image from DND dataset [44].”
Generating Training Data for Denoising Real RGB Images via Camera Pipeline Simulation (2019, methodology): “Since the focus of this work is not the network architecture, we used the author’s implementa-Table [26], which has been shown to achieve state of the art result in de-noising.”
Deep Plug-And-Play Super-Resolution for Arbitrary Blur Kernels (2019, methodology): “While achieving great success on bicubic degradation [36, 44, 68], these methods perform poorly on most of real images due to the mismatch of degradation models.”