DSAC — Differentiable RANSAC for Camera Localization

By Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, Carsten Rother. Computer Vision and Pattern Recognition, 2016

715 citations recorded by the source corpus.

RANSAC is an important algorithm in robust optimization and a central building block for many computer vision applications. In recent years, traditionally hand-crafted pipelines have been replaced by deep learning pipelines, which can be trained in an end-to-end fashion. However, RANSAC has so far not been used as part of such deep learning pipelines, because its hypothesis selection procedure is non-differentiable. In this work, we present two different ways to overcome this limitation. The most promising approach is inspired by reinforcement learning, namely to replace the deterministic hypothesis selection by a probabilistic selection for which we can derive the expected loss w.r.t. to all learnable parameters. We call this approach DSAC, the differentiable counterpart of RANSAC. We apply DSAC to the problem of camera localization, where deep learning has so far failed to improve on traditional approaches. We demonstrate that by directly minimizing the expected loss of the output camera poses, robustly estimated by RANSAC, we achieve an increase in accuracy. In the future, any deep learning pipeline can use DSAC as a robust optimization component.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/56ba6dfa111a3308ff5311997bb83a481953540e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, neutral): “In [60], [61], and [62], the authors introduce a novel differentiable way to apply RANSAC via sharing of hypotheses based on the predicted distribution.”
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions (2018, methodology): “They learn features with stable appearance over time [17, 47, 49], train classifiers for place recognition [12, 28, 37, 76], and train CNNs to regress 2D-3D matches [8, 9, 60] or camera poses [19, 31, 74].”
Learning to Find Good Correspondences (2017, methodology): “Dense methods aside, DSAC [4] recently introduced a differentiable outlier rejection scheme for monocular pose estimation.”
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis (2019, methodology): “DSAC [12] obtains 2D3D correspondences using a CNN architecture specifically TABLE 3 Comparison With State-of-the-Art Localization Methods on the InLoc Dataset”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “For trainable fundamental matrix estimation, Brachmann et al. (2017) proposed a differentiable RANSAC, termed as DSAC, which is based on reinforcement learning in an endto-end manner.”
DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “Using a collection of pointmaps as shape representation is quite counter-intuitive for MVS, but its usage is widespread for Visual Localization tasks, either in scene-dependent optimization approaches [8, 9, 11] or scene-agnostic inference methods [96, 124, 159].”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “Alternatively, machine learning techniques can be used to directly regress 3D point positions from image patches [8,10,16,43,46,47,65].”
A review of multimodal image matching: Methods and applications (2021, background): “For example, a differentiable RANSAC, namely, DSAC [338], is trained in an end-to-end manner by using reinforcement learning, while other efforts are made to improve the sampling strategy [339,340].”
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning (2018, methodology): “Subsequently a differentiable RANSAC [7] approach is used to get a camera pose that aligns to the predicted scene coordinates.”
Global Pose Estimation with an Attention-Based Recurrent Network (2018, methodology): “RANSAC [13] is commonly used for geometric verification and outlier rejection, and there are also prioritized feature matching approaches [39].”
Privacy Preserving Image Queries for Camera Localization (2019, background): “Alternatively, learning-based methods [13, 30, 71, 73] partially avoid the privacy issues associated with sharing confidential 3D point clouds, as they do not explicitly store 3D models.”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “In previous work, we have shown that the RANSAC-based pose optimization can be made differentiable to allow for end-to-end training of a scene coordinate regression pipeline [14], [15].”
Geometry-Aware Learning of Maps for Camera Localization (2017, methodology): “[7] localize a camera in a dense 3D reconstruction by performing RANSAC on predicted 2D-3D correspondences.”
Image-Based Localization Using LSTMs for Structured Feature Correlation (2016, background): “DSAC [4] proposes a differentiable RANSAC so that a matching function that optimizes pose quality can be learned.”
Learning Two-View Correspondences and Geometry Using Order-Aware Network (2019, methodology): “In the deep learning era, DSAC [2] mimicked the behavior of RANSAC and proposed a differentiable counterpart using probabilistic selection.”
Privacy Preserving Image-Based Localization (2019, methodology): “Traditional localization methods based on image retrieval [34,67] and based on learning [12, 35, 81, 82] have the advantage of not requiring the explicit storage of 3D maps.”
StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction (2018, methodology): “While this technique has roots in policy gradient approaches in reinforcement learning [57], it was recently formulated as stochastic computation graphs in [50] and applied to RANSAC-based camera localization in [5].”
Learning Less is More - 6D Camera Localization via 3D Surface Regression (2017, methodology): “Our approach is most closely related to the DSAC pipeline [2] which was introduced in detail above.”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “Common approaches encode the scene into a deep network by regressing from an input image to an absolute pose [35, 37, 59, 66, 90] or 3D scene coordinates [9, 13, 16, 17, 80].”
Cross-View Semantic Segmentation for Sensing Surroundings (2019, methodology): “sk is still at pure image level without providing any information about the spatial structure of the surroundings explicitly. There are many attempts to extract spatial knowledge from the image input [8, 3, 6, 7], however, 1Code and demo video are available at the project page: https:// view-parsing”