Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images

By Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, Andrew William Fitzgibbon. 2013

1,070 citations recorded by the source corpus.

We address the problem of inferring the pose of an RGB-D camera relative to a known 3D scene, given only a single acquired image. Our approach employs a regression forest that is capable of inferring an estimate of each pixel's correspondence to 3D points in the scene's world coordinate frame. The forest uses only simple depth and RGB pixel comparison features, and does not require the computation of feature descriptors. The forest is trained to be capable of predicting correspondences at any pixel, so no interest point detectors are required. The camera pose is inferred using a robust optimization scheme. This starts with an initial set of hypothesized camera poses, constructed by applying the forest at a small fraction of image pixels. Preemptive RANSAC then iterates sampling more pixels at which to evaluate the forest, counting inliers, and refining the hypothesized poses. We evaluate on several varied scenes captured with an RGB-D camera and observe that the proposed technique achieves highly accurate relocalization and substantially out-performs two state of the art baselines.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

RGB-D datasets using microsoft kinect or similar sensors: a survey (2016, methodology): “The curves under the images are the accelerometer data at 50 Hz of devices attached to the knife, the mixing spoon, the small spoon, the peeler, the glass, the oil bottle, and the pepper dispenser mentioned datasets above, another 26 extra RGB-D datasets for different applications are also added int”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “These previous RGB-D datasets have been used to train models for several standard scene understanding tasks, including semantic segmentation [27, 33, 13, 40, 7], 3D object detection [31, 24, 14, 36, 37], normal estimation [41, 23, 9, 3], camera relocalization [30, 39], and others [47, 11, 12].”
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2017, methodology): “It adopts the pre-emptive RANSAC algorithm as in [21] and [11] to handle noise in the results predicted from the network.”
SceneNN: A Scene Meshes Dataset with aNNotations (2016, methodology): “Our scenes could also be used for applications such as relocalization [26].”
Data-driven fluid simulations using regression forests (2015, methodology): “…and Object Modeling Physically based modeling; I.2.6 [Artificial Intelligence]: Learning; Keywords: fluid simulation, data-driven, regression forest ∗The authors assert equal contribution and joint first authorship †e-mail:lubor.ladicky,sohyeon.jeong,solenthaler,marc.pollefeys,grossm@inf.ethz.ch”
Uncertainty-Driven 6D Pose Estimation of Objects and Scenes from a Single RGB Image (2016, methodology): “[23] introduced the concept of scene coordinates for camera localization.”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, methodology): “Both PBRS and ‘Ours’ are trained on synthetic data and NYUv2 [27] (real data), and then tested on NYUv2, 7scenes [35], Scannet [5] and synthetic CG-PBR datasets.”
Feature-Metric Registration: A Fast Semi-Supervised Approach for Robust Point Cloud Registration Without Correspondences (2020, background): “7Scene [20] is a widely used benchmark registration dataset by a Kinect RGB-D camera on the indoor environment.”
Geometric Loss Functions for Camera Pose Regression with Deep Learning (2017, methodology): “Table 3: Median localization results for the Cambridge Landmarks [22] and 7 Scenes [46] datasets.”
Metric Regression Forests for Correspondence Estimation (2015, methodology): “Instead of assuming a uni-modal Gaussian distribu- 203 tion (e.g. Shotton et al. 2013) we use KDE to approximate the 204 density where the kernels are functions of the metric chosen; 205 see Fig.”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, methodology): “Critically, all top-performing learnable approaches are at least trained per-dataset, if not per-scene, and are limited to small environments [37, 80].”
Geometry-Aware Learning of Maps for Camera Localization (2017, methodology): “We evaluate the proposed methods extensively on both the indoor 7-Scenes [48] and the outdoor Oxford RobotCar dataset [38].”
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization (2015, methodology): “Our work most closely follows from the Scene Coordinate Regression Forests for relocalization proposed in [20].”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, methodology): “Alternatively, machine learning techniques can be used to directly regress 3D point positions from image patches [8,10,16,43,46,47,65].”
PREDATOR: Registration of 3D Point Clouds with Low Overlap (2020, methodology): “3DMatch: [13] is a collection of 62 scenes, combining earlier data from Analysis-by-Synthesis [10], 7Scenes [9], SUN3D [11], RGB-D Scenes v.”
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “We perform the experiments on two indoor datasets, ScanNet (V2) [8] and 7-Scenes [39].”