Discriminative Learning of Deep Convolutional Feature Point Descriptors

By Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, Francesc Moreno-Noguer. 2015

817 citations recorded by the source corpus.

Deep learning has revolutionalized image-level tasks such as classification, but patch-level tasks, such as correspondence, still rely on hand-crafted features, e.g. SIFT. In this paper we use Convolutional Neural Networks (CNNs) to learn discriminant patch representations and in particular train a Siamese network with pairs of (non-)corresponding patches. We deal with the large number of potential pairs with the combination of a stochastic sampling of the training set and an aggressive mining strategy biased towards patches that are hard to classify. By using the L2 distance during both training and testing we develop 128-D descriptors whose euclidean distances reflect patch similarity, and which can be used as a drop-in replacement for any task involving SIFT. We demonstrate consistent performance gains over the state of the art, and generalize well against scaling and rotation, perspective transformation, non-rigid deformation, and illumination changes. Our descriptors are efficient to compute and amenable to modern GPUs, and are publicly available.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Spread-Out Local Feature Descriptors (2017, methodology): “In the experiment section, we test combining the global orthogonal regularization with contractive loss3 (1) [19], triplet loss (2) [1], lifted structured similarity softmax loss (LSSS) [14] and N-pair loss [21].”
Set2Graph: Learning Graphs From Sets (2020, methodology): “…learning Previous models suggested for learning pairwise relations in sets were mostly of the form H 2 ( X ) = m ( φ ( x i ) , φ ( x j )) (Hsu et al., 2017; Bromley et al., 1994; Simo-Serra et al., 2015; Zagoruyko & Komodakis, 2015; Bell & Bala, 2015; Ahmed et al., 2015; Vo & Hays, 2016).”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “With the recent success of neural networks, several works have begun to explore the use of deep learning techniques for training state-of-the-art keypoint descriptors that can facilitate robust matching between keypoints and their local image features [45, 34, 16].”
Sampling Matters in Deep Embedding Learning (2017, background): “For the contrastive loss it is common to select from all posible pairs at random [3, 6, 11], and sometimes with hard negative mining [27].”
Fully-Convolutional Siamese Networks for Object Tracking (2016, methodology): “Deep Siamese conv-nets have previously been applied to tasks such as face verification [14,18,20], keypoint descriptor learning [19,21] and one-shot character recognition [22].”
Rotation Equivariant Vector Field Networks (2016, methodology): “We have presented a new way of hard-coding rotation equivariance in CNNs by applying each ﬁlter at different orientations and extracting a vector ﬁeld feature map that encodes the maximum activation magnitude and angle.”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, background): “2006), learning image descriptors (Simo-Serra et al. 2015), and performing face identification (Chopra et al.”
Generic 3D Representation via Pose Estimation and Matching (2016, background): “Therefore, we presume (and empirically observe) that the generality of our 3D representation is mostly attributed to the camera pose estimation component. scribing, and matching image features, either through a handcrafting the feature [35,11,39,38,5,40] or learning it [50,25,65,9,66,49,19].”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “DeepDesc (Simo-Serra et al. 2015) uses CNNs to learn discriminant patch representations together with L2 distance measuring.”
Deep Discrete Flow (2016, background): “A number of approaches [45, 46] aim for descriptor learning for sparse feature matching.”
Improving Transferability of Adversarial Examples With Input Diversity (2018, background): “These transformed adversarial examples are known as hard examples [29, 30] for attackers, which can then be served as good samples to produce more transferable adversarial examples.”
Adaptive Triplet Model for Fine-Grained Visual Categorization (2018, methodology): “Reference [34] proposes to use hard mining to focus on parts that are hard to classify.”
Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning (2017, positive): “Our approach is related to deep learning techniques that learn similarity metrics directly from images, such as siamese [16] and triplet networks [17].”
Comparative Evaluation of Hand-Crafted and Learned Local Features (2017, methodology): “[42] use a siamese architecture [5] with a 3-layer CNN to minimize the contrastive loss from Eq.”
Deep Metric Learning with Hierarchical Triplet Loss (2018, background): “…understanding tasks, including images or object retrieval [26,30,34], single-shot object classiﬁcation [30,34,32], key-point descriptor learning [12,24], face veriﬁcation [22,20], person re-identiﬁcation [30,23], object tracking [29] and etc. Recently, there is a number of widely-used loss…”
Neural network-based clustering using pairwise constraints (2015, background): “…learning (Chopra et al., 2005), dimensionality reduction (Hadsell et al., 2006), semi-supervised embedding (Weston et al., 2008), and some applications to image data, such as in learning to match patches (Han et al., 2015; Zagoruyko & Komodakis, 2015) and feature points (Simo-Serra et al., 2015).”