Learning to Find Good Correspondences

By Kwang Moo Yi, Eduard Trulls, Y. Ono, Vincent Lepetit, Mathieu Salzmann, Pascal Fua. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

529 citations recorded by the source corpus.

We develop a deep architecture to learn to find good correspondences for wide-baseline stereo. Given a set of putative sparse matches and the camera intrinsics, we train our network in an end-to-end fashion to label the correspondences as inliers or outliers, while simultaneously using them to recover the relative pose, as encoded by the essential matrix. Our architecture is based on a multi-layer perceptron operating on pixel coordinates rather than directly on the image, and is thus simple and small. We introduce a novel normalization technique, called Context Normalization, which allows us to process each data point separately while embedding global information in it, and also makes the network invariant to the order of the correspondences. Our experiments on multiple challenging datasets demonstrate that our method is able to drastically improve the state of the art with little training data.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/901e8381aae4b1fbb0d4dcef714d39fbf02f9681

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “Researchers have also used these for correct match selection and geometrical transformation model estimation from putative match sets (Moo Yi et al. 2018; Ma et al. 2019a; Zhao et al. 2019; Ranftl and Koltun 2018; Poursaeed et al. 2018).”
A review of multimodal image matching: Methods and applications (2021, background): “In general, parameter regression and inlier/outlier classification are trained jointly for performance enhancement [341].”
Deep Global Registration (2020, methodology): “by a number of learning-based methods for estimating the validity of correspondences in 2D [45, 33] and 3D [30].”
Learning Two-View Correspondences and Geometry Using Order-Aware Network (2019, methodology): “Finally, an essential matrix can be recovered from the inlier correspondences by a closed-form solution [15, 20].”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “, local features [63,79], outlier filtering [49,70], or scene coordinate regression [10,46], these approaches aim to learn the full localization pipeline.”
DISK: Learning local features with policy gradient (2020, methodology): “Note that we only compare with submissions using the built-in feature matcher, based on the l2 distance between descriptors, instead of neural-network based matchers [46, 49, 33], which combined with state-of-the-art features obtain the best overall results.”
LightGlue: Local Feature Matching at Light Speed (2023, methodology): “Those are filtered out by heuristics, like Lowe’s ratio test [41] or the mutual check, inlier classifiers [44, 82], and by robustly fitting geometric models [22, 7].”
PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency (2021, methodology): “Specifically, we compare (1) PointCN (3D version of [46], which is the feature extraction module adopted by 3DRegNet [48]); (2) Nonlocal (the SCNonlocal module without the spatial term, i.”
SuperGlue: Learning Feature Matching With Graph Neural Networks (2019, methodology): “Instance Normalization [54]: Attention, as used by SuperGlue, is a more flexible and powerful context aggregation mechanism than instance normalization, which treats all keypoints equally, as used by previous work on feature matching [30, 63, 29, 41, 6].”
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (2019, background): “More recently, [31] introduced a network to learn to find good correspondences for wide-baseline stereo.”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “Alternatively to making RANSAC differentiable, some authors propose to replace RANSAC by a neural network [61], [62], [63], [64], [65].”
LF-Net: Learning Local Features from Images (2018, background): “Some can only train individual parts of the feature extraction pipeline [33] while others can be trained end-to-end but still require the output of hand-crafted detectors to initialize the training process [11, 48, 49].”
Image Matching Across Wide Baselines: From Paper to Practice (2020, methodology): “It often works best when paired with RANSAC [112,95], so we consider it as an optional pre-filtering step before RANSAC – and apply it to both”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “Recently, many of these components have been learned with great success [2, 23, 25, 58, 60, 65, 67, 71, 95], but often independently and not end-to-end due to the complexity of such systems.”
ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning (2020, methodology): “This has changed over the past few years, with the arrival of deep learning and an abundance of new dense [57, 47, 60] and sparse [55, 12, 39, 59, 27] methods.”
Neural Nearest Neighbors Networks (2018, methodology): “For training we use the publicly available code of [48].”