Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

By Fangchang Ma, Guilherme Venturelli Cavalheiro, Sertac Karaman. IEEE International Conference on Robotics and Automation, 2018

460 citations recorded by the source corpus.

Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels for training. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) input to dense depth prediction. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that the self-supervised framework outperforms a number of existing solutions trained with semi-dense annotations. Furthermore, when trained with semi-dense annotations, our network attains state-of-the-art accuracy and is the winning approach on the KITTI depth completion benchmark1 at the time of submission.1http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/0d549872ded0fb6ab470a3073d42e0d87cd92fed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Learning Guided Convolutional Network for Depth Completion (2019, methodology): “2) Fusion Scheme of GuideNet and DepthNet: As described in subsection III-D, instead of using early or late feature fusion like existing methods [6], [7], [34], our approach fuses the decoder features of the GuideNet to the encoder features of the DepthNet.”
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, methodology): “We also employ the same basic training procedure as Ma et al. [32] to train all our models, see Appendix C.1 for details.”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “Qualitative comparison with some competing methods [33, 6] are shown in Fig.”
Learning Depth with Convolutional Spatial Propagation Network (2018, background): “Our strategy has several advantages over the previous stateof-the-art sparse-to-dense methods [9], [10], [71].”
FastDepth: Fast Monocular Depth Estimation on Embedded Systems (2019, background): “Fusion of RGB images and sparse depth measurements [24, 25] at early stages also improved the accuracy of depth estimation.”