Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT

By Philipp Fischer, Alexey Dosovitskiy, Thomas Brox. 2014

279 citations recorded by the source corpus.

Latest results indicate that features learned via convolutional neural networks outperform previous descriptors on classification tasks by a large margin. It has been shown that these networks still work well when they are applied to datasets or recognition tasks different from those they were trained on. However, descriptors like SIFT are not only used in recognition but also for many correspondence problems that rely on descriptor matching. In this paper we compare features from various layers of convolutional neural nets to standard SIFT descriptors. We consider a network that was trained on ImageNet and another one that was trained without supervision. Surprisingly, convolutional neural networks clearly outperform SIFT on descriptor matching. This paper has been merged with arXiv:1406.6909

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Discriminative Unsupervised Feature Learning with Convolutional Neural Networks (2014, background): “For example, unsupervised feature learning is known to be beneficial for image restoration [6] and recent results show that it outperforms supervised feature learning also on descriptor matching [7].”
Learning to See by Moving (2015, background): “The aim of such approaches has been to discover compact and rich representations of images that are also sufﬁcient to reconstruct these images.”
FlowNet: Learning Optical Flow with Convolutional Networks (2015, background): “[12] extract feature representations from CNNs trained in supervised or unsupervised manner and match these features based on Euclidean distance.”
LSDA: Large Scale Detection through Adaptation (2014, background): “, subcategory classification, scene classification, domain adaptation [8] and even image matching [9].”
Learning to compare image patches via convolutional neural networks (2015, methodology): “[10] analysed the performance of convolutional descriptors from AlexNet network (that was trained on Imagenet dataset [13]) on the wellknown Mikolajczyk dataset [16] and showed that these convolutional descriptors outperform SIFT in most cases except blur.”
Generic 3D Representation via Pose Estimation and Matching (2016, background): “Therefore, we presume (and empirically observe) that the generality of our 3D representation is mostly attributed to the camera pose estimation component. scribing, and matching image features, either through a handcrafting the feature [35,11,39,38,5,40] or learning it [50,25,65,9,66,49,19].”
Learning local feature descriptors with triplets and shallow convolutional neural networks (2016, methodology): “Interest in the field started from results shown in [9] that the features from the last layer of a convolutional deep network trained on ImageNet [18] can outperform SIFT.”
Convolutional Neural Network-based Place Recognition (2014, methodology): “In [Fischer, et al., 2014], features from various layers of CNNs are evaluated and compared with SIFT descriptors on a descriptor matching benchmark.”
Neural best-buddies (2018, methodology): “More recently, various CNN-based descriptors were offered to replace traditional gradient based ones [Fischer et al. 2014].”
Multi-view Harmonized Bilinear Network for 3D Object Recognition (2018, background): “Meanwhile, recent studies [6, 21] show the local convolutional feature from convolutional neural network is a natural and effective representation for the local patch, making it feasible to integrate the patch-based method in the network.”
Fully Convolutional Networks for Semantic Segmentation (2017, background): “…Motivation Network Architecture Fully convolutional networks Skip layers Results Summary PAGE 2 Use convnets to make pixel-wise predictions Semantic segmentation provides the “what” and “where” Does not require Patchwise training Refinement by superpixel projection, random field…”
Do Convnets Learn Correspondence? (2014, background): “Szegedy et al. [25] show counterintuitive properties of the convnet representation, and suggest that individual feature channels may not be more semantically meaningful than other bases in feature space.”
Visual Instance Retrieval with Deep Convolutional Networks (2014, background): “Much recent work has illuminated the ﬁeld on how to design and train Con-vNets to maximize performance 1)5) and also how to exploit learned ConvNet representations to solve visual recognition tasks 7)9)11) in the sense of transfer learning.”
Taking a deeper look at pedestrians (2015, background): “Such architectures currently provide top results for general object classification [27, 38], general object detection [44], feature matching [17], stereo matching [51], scene recognition [55, 8], pose estimation [47, 7], action recognition [25, 42] and many other tasks [37, 2].”
Neighbourhood Consensus Networks (2018, unknown): “…of trainable image descriptors are based on convolutional neural networks (CNNs) and typically operate on patches extracted using a feature detector such as DoG [23], yielding a sparse set of descriptors [17, 10, 3, 37, 36, 4] or use a pre-trained image-level CNN feature extractor [26, 32].”
Visual Representations: Defining Properties and Deep Approximations (2014, background): “ferred from data, leading to representation learning [46,35,47,8,53,4]. There are more local descriptors than we can review here, and empirical comparisons [41] have recently expanded to include CNNs [24]. Unfortunately, many details and parameters make it hard to draw general conclusions [10] let a”