R2D2: Repeatable and Reliable Detector and Descriptor

By Jérôme Revaud, Philippe Weinzaepfel, C. D. Souza, No'e Pion, Gabriella Csurka, Yohann Cabon, Martin Humenberger. arXiv.org, 2019

540 citations recorded by the source corpus.

Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/911aa843612a8a47579e4d3da141fdee564965ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

LoFTR: Detector-Free Local Feature Matching with Transformers (2021, methodology): “0m,10°) Local Feature Evaluation on Night-time Queries R2D2 [32]+NN 71.”
LightGlue: Local Feature Matching at Light Speed (2023, methodology): “While classical algorithms rely on hand-crafted criteria and gradient statistics [41, 23, 4, 53], much of the recent research has focused on designing Convolutional Neural Networks (CNNs) for both detection [81, 16, 19, 52, 73] and description [42, 72].”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, background): “Learning-based sparse feature pipelines [41], [42], [43], [44] might ultimately be able to overcome these issues, but currently it is an open research question whether learned sparse features consistently exceed the capabilities of their hand-crafted predecessors [45], [46].”
DISK: Learning local features with policy gradient (2020, methodology): “Others have developed end-to-end-trainable pipelines [45, 10, 29, 11, 31] that can optimize the whole process and, hopefully, improve performance.”
PREDATOR: Registration of 3D Point Clouds with Low Overlap (2020, methodology): “Fully convolutional architectures [22] that enable dense feature computation over the whole input in a single forward pass [11, 12, 29] have been adopted to design faster 3D feature descriptors.”
Learning and aggregating deep local descriptors for instance-level recognition (2020, background): “Learning is performed with or without labeling at the local level in a number of diﬀerent approaches [38,13,12,7,10,47].”
ASLFeat: Learning Local Features of Accurate Shape and Localization (2020, methodology): “Due to the increasing prevalence of the joint learning with keypoint detectors [7, 27, 29, 8, 5], recent research focus has shifted to frameworks that densely extract features from image inputs , whereas no pre-deﬁned keypoint is given and thus previous patch-wise shape estimation becomes…”
Unifying Deep Local and Global Features for Image Search (2020, methodology): “propose to train the model using only image-level labels, as illustrated in Fig.2. In particular, note that we do not require patch-level supervision to train local features, unlike most recent works [14,49,36,29]. Besides the challenge to acquire the annotations, note that patch-level supervision c”
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer (2022, methodology): “As is shown in previous works [10,8,9,12,28,27,26,29], CNN exhibits strong capability to capture local context and generates high-level features, which can be directly used to perform nearest neighbour matching.”
D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features (2020, background): “In 2D image matching, several works have tackled the keypoint detection and description problems jointly [35, 6, 20, 24, 3, 26, 19].”
SuperGlue: Learning Feature Matching With Graph Neural Networks (2019, methodology): “Recent works on deep learning for matching often focus on learning better sparse detectors and local descriptors [16, 17, 34, 42, 61] from data using Convolutional Neural Networks (CNNs).”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “In addition, many methods integrate feature detection into the entire matching pipeline by jointly training with feature description and matching (Yi et al. 2016; DeTone et al. 2018; Ono et al. 2018; Shen et al. 2019; Dusmanu et al. 2019; Choy et al. 2016; Rocco et al. 2018; Dusmanu et al. 2019; Rev”
Grounding Image Matching in 3D with MASt3R (2024, methodology): “Departing from the former handcrafted methods like SIFT [52,71], modern approaches have been shifting towards learning-based data-driven schemes for detecting keypoints [8,60,97,117], describing them [7,33, 37,88] or both at the same time [10,21,53,54,70,98].”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “Recently, many of these components have been learned with great success [2, 23, 25, 58, 60, 65, 67, 71, 95], but often independently and not end-to-end due to the complexity of such systems.”
COTR: Correspondence Transformer for Matching Across Images (2021, background): “Learned feature extractors became popular with the introduction of LIFT [74], with many follow-ups [13, 44, 16, 49, 5, 71].”