Back to the Feature: Learning Robust Camera Localization from Pixels to Pose

By Paul-Edouard Sarlin, Ajaykumar Unagar, Maans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl and 1 more. Computer Vision and Pattern Recognition, 2021

331 citations recorded by the source corpus.

Camera pose estimation in known scenes is a 3D geometry task recently tackled by multiple learning algorithms. Many regress precise geometric quantities, like poses or 3D points, from an input image. This either fails to generalize to new viewpoints or ties the model parameters to a specific scene. In this paper, we go Back to the Feature: we argue that deep networks should focus on learning robust and invariant visual features, while the geometric estimation should be left to principled algorithms. We introduce PixLoc, a scene-agnostic neural network that estimates an accurate 6-DoF pose from an image and a 3D model. Our approach is based on the direct alignment of multiscale deep features, casting camera localization as metric learning. PixLoc learns strong data priors by end-to-end training from pixels to pose and exhibits exceptional generalization to new scenes by separating model parameters and scene geometry. The system can localize in large environments given coarse pose priors but also improve the accuracy of sparse feature matching by jointly refining keypoints and poses with little overhead. The code will be publicly available at github.com/cvg/pixloc.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/2521e94c99bc247e81c4ab9330641662d2f8f501

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Jia Deng
2. Zachary Teed
3. Paul-Edouard Sarlin
4. Marc Pollefeys
5. Philipp Lindenberger
6. Michael Felsberg
7. Christian Häne
8. Jia-Bin Huang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Andrea Vedaldi
4. Dacheng Tao
5. Jia Deng
6. Jitendra Malik
7. Xiaogang Wang
8. Chunhua Shen

In Geology

1. Jia Deng
2. Zachary Teed
3. Marc Pollefeys
4. Philipp Lindenberger
5. Paul-Edouard Sarlin
6. Abhinav Kumar Gupta

Citations in the corpus

• LightGlue: Local Feature Matching at Light Speed (2023, background): “SuperGlue is highly effective for visual localization in challenging conditions [59, 55, 58, 57] and generalizes well to other tasks like aerial matching [83], object pose estimation [69], and even fish re-identification [47].”
• DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “Our method obtains comparable accuracy compared to existing approaches, being feature-matching ones [101, 103] or end-to-end learning-based methods [11, 55, 102, 125, 152], even managing to out-perform strong baselines like HLoc [101] in some cases.”
• RoMa: Robust Dense Feature Matching (2023, positive): “Interestingly, we find that the VGG19 features are worse than the ResNet features for coarse matching, despite VGG feature being widely used as local features [16, 42, 52].”
• DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, background): “Many works have focused on training systems for particular subproblems, such as feature detection [13, 7, 29, 26, 35], feature matching and outlier rejection [38, 36], and localization [51, 39].”