Working hard to know your neighbor's margins: Local descriptor learning loss

By Anastasiia V. Mishchuk, Dmytro Mishkin, Filip Radenovic, Jiri Matas. 2017

799 citations recorded by the source corpus.

We introduce a novel loss for learning local feature descriptors which is inspired by the Lowe's matching criterion for SIFT. We show that the proposed loss that maximizes the distance between the closest positive and closest negative patch in the batch is better than complex regularization methods; it works well for both shallow and deep convolution network architectures. Applying the novel loss to the L2Net CNN architecture results in a compact descriptor -- it has the same dimensionality as SIFT (128) that shows state-of-art performance in wide baseline stereo, patch verification and instance retrieval benchmarks. It is fast, computing a descriptor takes about 1 millisecond on a low-end GPU.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Spread-Out Local Feature Descriptors (2017, background): “Recently, there are a lot of works such as smart sampling strategies [1, 12] and structured loss [23] that improve the triplet loss.”
From Coarse to Fine: Robust Hierarchical Localization at Large Scale (2018, background): “Additionally, SIFT is not competitive with recent learned features, especially under large illumination changes [16, 36, 12, 32].”
Unifying Deep Local and Global Features for Image Search (2020, background): “Recently, several deep learning-based local features have been proposed [63,39,33,42,34,49,14,6,29].”
SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration (2020, background): “The widelyused hardest in batch sampling [42] is also adopted on-thefly to maximize the distance between the closest positive and the closest negative patches.”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, methodology): “Most deep feature matching meth-ods have focused on learning the descriptor component, applied either on a sparse set of keypoints [3, 51, 52, 29] detected using standard handcrafted methods or densely over the image [12, 47, 54, 32].”
COTR: Correspondence Transformer for Matching Across Images (2021, methodology): “Popular patch descriptors range from hand-crafted [34, 9] to learned [42, 66, 17] ones.”
Learning With Average Precision: Training Image Retrieval With a Listwise Loss (2019, background): “First, it requires a number of ad hoc heuristics such as pre-training for classification [1, 18], combining several losses [10, 12] and biasing the sampling of image pairs by mining hard or semi-hard negative examples [15, 21, 37, 38].”
LightGlue: Local Feature Matching at Light Speed (2023, methodology): “While classical algorithms rely on hand-crafted criteria and gradient statistics [41, 23, 4, 53], much of the recent research has focused on designing Convolutional Neural Networks (CNNs) for both detection [81, 16, 19, 52, 73] and description [42, 72].”
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features (2019, methodology): “To this end, we propose an extension to the triplet margin ranking loss, which has been successfully used for descriptor learning [6, 34], to also account for the detection stage.”
Features for Multi-target Multi-camera Tracking and Re-identification (2018, background): “Hermans et al [35] and Mischuk et al [54] have proposed the batch-hard triplet loss with built-in hard sample mining.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “HardNet (Mishchuk et al. 2017) achieves better improvement than L2Net by using a simple hinge triplet loss with the “hardest-within-batch” mining.”
DISK: Learning local features with policy gradient (2020, methodology): “While patch descriptors can be easily learned on predefined keypoints [38, 39, 25, 40, 13], joint detection and matching is harder to relax in a differentiable manner, due to its computational complexity.”
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location (2020, methodology): “We use HardNet [70] and use FPR (false positive rate) at 0.95 true positive recall as an evaluation metric (lower is better).”
Mining on Manifolds: Metric Learning Without Labels (2018, background): “Simple hard negative mining from a different class label might corrupt the training process due to mislabeled images [47, 33].”
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (2019, methodology): “We compute the covariant constraint loss LM-SIP for five scale levels, with the size of the M-SIP windowsNs ∈ [8, 16, 24, 32, 40] and loss term λs ∈ [256, 64, 16, 4, 1], that were determined by performing a hyperparameter search on the validation”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “Many works have focused on training systems for particular subproblems, such as feature detection [13, 7, 29, 26, 35], feature matching and outlier rejection [38, 36], and localization [51, 39].”
ASLFeat: Learning Local Features of Accurate Shape and Localization (2020, methodology): “We compare our methods with 1) patch descriptors, including HardNet++ [21] with SIFT [17] detector (SIFT + HN++), or plus a shape estimator HesAffNet [22] (HAN + HN++).”