Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

By Fangchang Ma, Sertac Karaman. IEEE International Conference on Robotics and Automation, 2017

667 citations recorded by the source corpus.

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59 % to 92 % on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software22https://github.com/fangchangma/sparse-to-dense and video demonstration33https://www.youtube.com/watch?v=vNIIT_M7×7Y are publicly available.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/483f26c0651bf9f57ce231293736ec5c4e6f54d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Depth with Convolutional Spatial Propagation Network (2018, methodology): “Nevertheless, upon a closer inspection of the outputs from a contemporary approach for depth completion [10] (Fig.”
Invertibility of Convolutional Generative Networks from Partial Measurements (2018, background): “Firstly, the inversion itself can be applied in image in-painting [21, 17], image reconstruction from sparse measurements [14, 13], and image manipulation [22] (e.”
Learning Guided Convolutional Network for Depth Completion (2019, methodology): “Following the similar setting of previous depth completion methods [6], [35], [54], our method is trained on 50 k images uniformly sampled from the training set, and tested on the 654 ofﬁcial labeled test set for evaluation.”
Non-Local Spatial Propagation Network for Depth Completion (2020, methodology): “Compared to geometry-agnostic methods [21, 19, 9], geometry-aware ones [14, 8, 25, 32] show better performance in general.”
Underwater Single Image Color Restoration Using Haze-Lines and a New Quantitative Dataset (2018, background): “5 cm, which is quite accurate considering the precision of existing datasets [44].”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “Adopting similar experiment setting as [6, 34], we synthetically generate sparse depth via random sampling, train on 50K images sampled from the training set, and evaluate on the official labeled test set (containing 654 images).”
Benchmarking Single-Image Dehazing and Beyond (2017, methodology): “For example, due to the limitations of RGBbased depth cameras, the Make3D dataset suffer from at least 4 meters of average root mean squared error in the predicted depths, and the KITTI dataset has at least 7 meters of average error [48].”
Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera (2018, background): “Karaman [10] proposed an end-to-end deep regression model for depth completion.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, methodology): “E STANDS FOR ROOT MEAN SQUARED ERROR [MM]. MAE STANDS FOR MEAN ABSOLUTE ERROR [MM]. Methods Fusion Level Type Models Run-time iRMSE iMAE RMSE MAE MonoLiDAR Fusion Signal Level Supervised Sparse2Dense [37] 0.08s 4.07 1.57 1299.85 350.32 Self-supervised Sparse2Dense++ [38] 0.08s 2.80 1.21 814.73 249.9”
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network (2018, background): “Nevertheless, upon closer inspection of the output from a contemporary approach [13] (Fig.”
Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries (2018, methodology): “1RMS values given in [28] are omitted here (displayed as (-)), because we found an error in the authors’ code, and believe that the numbers reported in their paper are miscalculated.”