Geometry-Aware Learning of Maps for Camera Localization

By Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, Jan Kautz. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

444 citations recorded by the source corpus.

Maps are a key component in image-based camera localization and visual SLAM systems: they are used to establish geometric constraints between images, correct drift in relative pose estimation, and relocalize cameras after lost tracking. The exact definitions of maps, however, are often application-specific and hand-crafted for different scenarios (e.g. 3D landmarks, lines, planes, bags of visual words). We propose to represent maps as a deep neural net called MapNet, which enables learning a data-driven map representation. Unlike prior work on learning maps, MapNet exploits cheap and ubiquitous sensory inputs like visual odometry and GPS in addition to images and fuses them together for camera localization. Geometric constraints expressed by these inputs, which have traditionally been used in bundle adjustment or pose-graph optimization, are formulated as loss terms in MapNet training and also used during inference. In addition to directly improving localization accuracy, this allows us to update the MapNet (i.e., maps) in a self-supervised manner using additional unlabeled video sequences from the scene. We also propose a novel parameterization for camera rotation which is better suited for deep-learning based camera pose regression. Experimental results on both the indoor 7-Scenes dataset and the outdoor Oxford RobotCar dataset show significant performance improvement over prior work. The MapNet project webpage is https://goo.gl/mRB3Au.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/8c0514cbf116afc4363152514b3ccedb05195a59

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

TartanAir: A Dataset to Push the Limits of Visual SLAM (2020, background): “Impressive progress has been made with geometric-based [2]–[5], learning-based [6]–[8], and hybrid methods [9]–[11].”
L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving (2019, background): “Improvements [6, 4] over PoseNet incorporate relative geometric constraints between frames or integrate temporal information through a bidirectional LSTM.”
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis (2019, methodology): “Wealso compare on our dataset to the state-of-the-art learning based pose estimators, DSAC [12] (and its improved version [13] (DSAC++)), PoseNet [40], and MapNet [15].”
Cross-View Semantic Segmentation for Sensing Surroundings (2019, methodology): “sk is still at pure image level without providing any information about the spatial structure of the surroundings explicitly. There are many attempts to extract spatial knowledge from the image input [8, 3, 6, 7], however, 1Code and demo video are available at the project page: https:// view-parsing”
From Coarse to Fine: Robust Hierarchical Localization at Large Scale (2018, background): “Some approaches directly regress the pose from a single image [6, 20], but are not competitive in term of accuracy [44].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “[14] and batch adaptation [13], [367], according to whether a single instance or a batch of instances exist at test time.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, methodology): “For evaluation, we use both PoseNet [29, 30] and MapNet [11].”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “[5] exploited geometric constraints from a diversity of sensory inputs for improving localization accuracy on a broad scale.”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “Absolute pose regression methods [11], [26], [27], [28], [29] aim at overcoming the precision limitation of image retrieval while preserving efficiency and scalability.”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, methodology): “Following [35], [36], [37], we use the logarithm of a unit quaternion to parameterize the rotation in R(3) and do not require an added normalization constraint unlike previous works [38].”
End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners (2018, background): “A trending topic is to combine digital maps and street-view images for accurate vehicle localization [58,59,60,61].”