N4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms

By Yaroslav Ganin, Victor S. Lempitsky. 2014

277 citations recorded by the source corpus.

We propose a new architecture for difficult image processing operations, such as natural edge detection or thin object segmentation. The architecture is based on a simple combination of convolutional neural networks with the nearest neighbor search. We focus our attention on the situations when the desired image transformation is too hard for a neural network to learn explicitly. We show that in such situations, the use of the nearest neighbor search on top of the network output allows to improve the results considerably and to account for the underfitting effect during the neural network training. The approach is validated on three challenging benchmarks, where the performance of the proposed architecture matches or exceeds the state-of-the-art.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Holistically-Nested Edge Detection (2015, methodology): “Here instead we use a simpler strategy to automatically balance the loss between positive/negative classes.”
Learning Representations for Automatic Colorization (2016, methodology): “Motivating this strategy is the success of similar architectures for semantic segmentation [4,10,14,23,27] and edge detection [1,11,25,34,43].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “Through these advances we train in an end-to-end manner a CNN that simultaneously addresses (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection.”
FlowNet: Learning Optical Flow with Convolutional Networks (2015, neutral): “Recent applications of CNNs include semantic segmentation [11, 15, 17, 28], depth prediction [10], keypoint prediction [17] and edge detection [13].”
DeepEdge: A multi-scale bifurcated deep network for top-down contour detection (2014, neutral): “While SCT [20] is a sparse coding approach, both N(4) fields [9] and DeepNet [16] use Convolutional Neural Networks (CNNs) to predict contours.”
Pushing the Boundaries of Boundary Detection using Deep Learning (2015, neutral): “Method ODS OIS AP gPb-owt-ucm (Arbelaez et al., 2011) 0.726 0.757 0.696 SE-Var (Doll´ar & Zitnick, 2015) 0.746 0.767 0.803 DeepNets (Kivinen et al., 2014) 0.738 0.759 0.758 N4-Fields (Ganin & Lempitsky, 2014) 0.753 0.769 0.784 DeepEdge (Bertasius et al., 2015) 0.753 0.772 0.807 CSCNN (Hwang & Liu, 2”
Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards (2016, background): “For example, Ning et al. (2005); Ciresan, Giusti, Gambardella, and Schmidhuber (2012); Pinheiro and Collobert (2013); and Ganin and Lempitsky (2014) use multilayered CNNs for image segmentation, in which individual patches representing contextual regions around pixels are densely classified in an…”
Learning Deep Representation for Imbalanced Classification (2016, methodology): “This is partially validated by the fact that while [27] and [35] need to train random forests on features to achieve competitive results, ours only relies on kNN label transfer as in [13], but with better results.”
Fully Convolutional Networks for Semantic Segmentation (2017, methodology): “By contrast, prior work randomly samples patches over a full dataset [10], [11], [12], [13], [16], potentially resulting in higher variance batches that may accelerate convergence [53].”
Semantic Segmentation with Boundary Neural Fields (2015, background): “…that our boundary-based affinity function is better suited for semantic segmentation than the common color-based affinities and 2) showing that our proposed global energy achieves better qualitative and quantitative semantic segmentation results in comparison to prior globalization models.”
Unsupervised Learning of Discriminative Attributes and Visual Representations (2016, methodology): “We simply transfer the neighboring contour patches to the output and average all the overlapping patches as in [19], followed by non-maximum suppression.”