Geometric Loss Functions for Camera Pose Regression with Deep Learning

By Alex Kendall, Roberto Cipolla. 2017

840 citations recorded by the source corpus.

Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet [22] is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single image. It learns to localize using high level features and is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treatment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to automatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNets performance across datasets ranging from indoor rooms to a small city.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps (2018, methodology): “PoseNet [25, 26] treats pose estimation as a regression problem based on a convolutional neural network.”
Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map (2017, neutral): “For space reason, we also leave out those (machine) learning-based methods ( e.g . regression forest [49], deep PoseNet [26, 27]).”
Egocentric Shopping Cart Localization (2018, background): “To balance the contribution of position prediction and orientation estimation, in [2] different loss functions are compared for the same network.”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, methodology): “An alternative is to train a convolutional neural network (CNN) to regress geometric quantities such as camera poses [5, 24, 35, 37, 42, 90, 97] or the 3D scene coordinate corresponding to each pixel [9–11, 13, 16, 17, 45, 80, 93].”
Geometry-Aware Learning of Maps for Camera Localization (2017, methodology): “Our work is built upon prior works in DNN-based pose estimation methods [11, 30, 31, 32, 40, 55], which regress 6-DoF camera pose from an input RGB image with a DNN.”
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis (2019, methodology): “We provide comparison with DSAC [12] and PoseNet [40] on much smaller datasets in Section 5.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, methodology): “For evaluation, we use both PoseNet [29, 30] and MapNet [11].”
DeepIM: Deep Iterative Matching for 6D Pose Estimation (2018, methodology): “(Kendall and Cipolla, 2017) proposed a geometric reprojection error as the loss function for pose regression that computes the average distance between the 2D projections of 3D points in the scene using the ground truth pose and the estimated pose.”
CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation (2019, background): “Direct approaches were initially proposed in some works on 6-DoF camera pose estimation and viewpoint estimation [22, 11, 10, 16, 23, 14].”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, methodology): “Following the PoseNet [48], we use the geometric matching loss for training, which avoids the balancing”
On the Continuity of Rotation Representations in Neural Networks (2018, methodology): “Csiszar et al. [8] used a 6D representations comprised of the sine and cosine of the three Euler angles, and Kendall et al. [16] used quaternions that are constrained to one hemisphere.”
Learning Less is More - 6D Camera Localization via 3D Surface Regression (2017, methodology): “Approaches like PoseNet [11] and its derivatives [29, 10] exhibit low localization accuracy so far.”
From Coarse to Fine: Robust Hierarchical Localization at Large Scale (2018, background): “Some approaches directly regress the pose from a single image [6, 20], but are not competitive in term of accuracy [44].”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “Besides video prediction, several studies analyzed the impact of different loss functions in image restoration [39], classification [40], camera pose regression [41] and structured prediction [42], among others.”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “Absolute pose regression methods [11], [26], [27], [28], [29] aim at overcoming the precision limitation of image retrieval while preserving efficiency and scalability.”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, background): “This loss is critical for good performance ([8], and see Section 5.”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, methodology): “[23] cast the 6-DoF camera pose relocalization problem as a learning task, and extended it upon the foundations of multiview geometry [22].”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, background): “Kendall et. al [20, 19] focused on the problem of camera localization, while other work [49, 42, 44, 48, 41] aim to estimate camera motion between a pair of frames.”