LF-Net: Learning Local Features from Images

By Y. Ono, Eduard Trulls, Pascal Fua, Kwang Moo Yi. Neural Information Processing Systems, 2018

594 citations recorded by the source corpus.

We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/70073a4d2ebbb7ce5f432075e60850855ef9d7f6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised label noise modeling and loss correction (2019, background): “Convolutional Neural Networks (CNNs) have recently become the par excellence base approach to deal with many computer vision tasks (DeTone et al., 2016; Ono et al., 2018; Beluch et al., 2018; Redmon et al., 2016; Zhao et al., 2017; Krishna et al., 2017).”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “Local features that are more robust to large viewpoint changes are an active field of research [51, 55] and structure-based methods will automatically benefit from progress in this field.”
D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features (2020, background): “In 2D image matching, several works have tackled the keypoint detection and description problems jointly [35, 6, 20, 24, 3, 26, 19].”
ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning (2020, methodology): “Finally, we report that our method also works well with two state-of-the-art, learned local feature methods – SuperPoint [14] and LF-Net [35].”
Unifying Deep Local and Global Features for Image Search (2020, background): “Recently, several deep learning-based local features have been proposed [63,39,33,42,34,49,14,6,29].”
A review of multimodal image matching: Methods and applications (2021, methodology): “For learning-based detectors, we choose TILDE [182], LFnet [188], SuperPoint [185] for evaluation.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, methodology): “Keypoint Correspondence methods aim at sparse keypoint matching given image pairs, from hand-defined discriminative feature descriptors [3, 36, 37, 57] to deep learning based methods [12, 29, 39, 45].”
Learning Two-View Correspondences and Geometry Using Order-Aware Network (2019, methodology): “With the emergence of deep learning, many works attempted to employ learning-based methods to solve geometric matching tasks, including both dense methods [36, 45, 13, 30] and sparse methods [40, 23, 4, 3, 18, 17].”
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (2019, methodology): “A number of related works have focused their objective function to address this issue, although they were based either on local patches [9, 10] or global map regression loss [13, 14, 11].”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, background): “Deep models [15, 40, 45, 52, 66, 71] learn effective features for dense correspondence that can be robust to photometric and geometric changes such as rotation, scaling, or perspective transformation.”
Segmentation-Driven 6D Object Pose Estimation (2018, background): “Over the years, much effort has been invested in designing local feature descriptors that are invariant to various transformations [27, 40, 41, 42, 34, 32], so that they can be matched more robustly [29, 30, 14].”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “…the interest points in a supervised (Yi et al. 2016; Verdie et al. 2015; Zhang et al. 2017b), self-supervised (Zhang and Rusinkiewicz 2018; DeTone et al. 2018), or unsupervised manner (Lenc and Vedaldi 2016; Savinov et al. 2017;Ono et al. 2018;Georgakis et al. 2018;Barroso-Laguna et al. 2019).”
ASLFeat: Learning Local Features of Accurate Shape and Localization (2020, methodology): “Due to the increasing prevalence of the joint learning with keypoint detectors [7, 27, 29, 8, 5], recent research focus has shifted to frameworks that densely extract features from image inputs , whereas no pre-deﬁned keypoint is given and thus previous patch-wise shape estimation becomes…”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Region-wise correspondence could be learned in a supervised [70,78,84], self-supervised [19, 40, 83], or unsupervised manner [3, 28, 35,52,61].”
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features (2019, methodology): “2a illustrates the common variations of this pipeline, from using hand-crafted detectors [7,19,29,31,33] and descriptors [7,9,24,29,44], replacing either the descriptor [6,54,55] or detector [49,68] with a learned alternative, or learning both the detector and descriptor [38,64].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Structure-From-Motion (SFM) [12, 30, 43, 48] relies on sparse keypoint matches given image pairs of rigid scenes, and typically ignores occlusion, as it can use geometry to filter errors [21, 55, 71, 73].”