Understanding the Limitations of CNN-Based Absolute Camera Pose Regression

By Torsten Sattler, Qunjie Zhou, Marc Pollefeys, L. Leal-Taix'e. Computer Vision and Pattern Recognition, 2019

440 citations recorded by the source corpus.

Visual localization is the task of accurate camera pose estimation in a known scene. It is a key problem in computer vision and robotics, with applications including self-driving cars, Structure-from-Motion, SLAM, and Mixed Reality. Traditionally, the localization problem has been tackled using 3D geometry. Recently, end-to-end approaches based on convolutional neural networks have become popular. These methods learn to directly regress the camera pose from an input image. However, they do not achieve the same level of pose accuracy as 3D structure-based methods. To understand this behavior, we develop a theoretical model for camera pose regression. We use our model to predict failure cases for pose regression techniques and verify our predictions through experiments. We furthermore use our model to show that pose regression is more closely related to pose approximation via image retrieval than to accurate pose estimation via 3D structure. A key result is that current approaches do not consistently outperform a handcrafted image retrieval baseline. This clearly shows that additional research is needed before pose regression algorithms are ready to compete with structure-based methods.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/8f1379c4f8901098a191e35c5543b312e569d68f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Eduard Trulls
2. Pascal Fua
3. Kwang Moo Yi
4. Carsten Rother
5. Marc Pollefeys
6. Eric Brachmann
7. M. Tyszkiewicz
8. Dmytro Mishkin

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Pascal Fua
3. Andrew Zisserman
4. Andrea Vedaldi
5. Dacheng Tao
6. Jitendra Malik
7. Marc Pollefeys
8. Ming-Hsuan Yang

In Geology

1. Jiri Matas
2. Pascal Fua
3. Carsten Rother
4. Eduard Trulls
5. Kwang Moo Yi
6. Eric Brachmann
7. Dmytro Mishkin
8. M. Tyszkiewicz

Citations in the corpus

• Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “Finally, approaches based on image retrieval or pose regression require only a database of RGB images and ground truth poses for re-localization, but suffer from low precision comparable to GPS [12].”
• Lending Orientation to Neural Networks for Cross-View Geo-Localization (2019, neutral): “, [5, 24, 19, 18, 20, 27]), using ground-to-aerial cross-view matching (i.”
• DISK: Learning local features with policy gradient (2020, background): “While not immune to the deep learning “revolution”, 3D reconstruction is one of the last bastions where sparse, hand-crafted solutions remain competitive with or outperform their dense, learned counterparts [37, 34, 16].”
• Image Matching Across Wide Baselines: From Paper to Practice (2020, background): “Despite their promise, they are yet to outperform the separatists [86,119] that are based on the classical paradigm of step-by-step solutions.”
• CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation (2019, neutral): “Direct approaches were initially proposed in some works on 6-DoF camera pose estimation and viewpoint estimation [22, 11, 10, 16, 23, 14].”