DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image

By Jiaxiong Qiu, Zhaopeng Cui, Yinda Zhang, Xingdi Zhang, Shuaicheng Liu, Bing Zeng, Marc Pollefeys. Computer Vision and Pattern Recognition, 2018

428 citations recorded by the source corpus.

In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth. Inspired by the indoor depth completion, our network estimates surface normals as the intermediate representation to produce dense depth, and can be trained end-to-end. With a modified encoder-decoder structure, our network effectively fuses the dense color image and the sparse LiDAR depth. To address outdoor specific challenges, our network predicts a confidence mask to handle mixed LiDAR signals near foreground boundaries due to occlusion, and combines estimates from the color image and surface normals with learned attention maps to improve the depth accuracy especially for distant areas. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on KITTI depth completion benchmark. Ablation study shows the positive impact of each model components to the final performance, and comprehensive analysis shows that our model generalizes well to the input with higher sparsity or from indoor scenes.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8b4765c33816247cfea7458c3664d162fc044534

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. In-So Kweon
2. Wanli Ouyang
3. Ming-Hsuan Yang
4. Luc Van Gool
5. Xiaogang Wang
6. Dacheng Tao
7. Ping Tan
8. Yi Yang

Citations in the corpus

• Non-Local Spatial Propagation Network for Depth Completion (2020, methodology): “AE iRMSE iMAE CSPN [9] 1019.64 279.46 2.93 1.15 DDP [33] 832.94 203.96 2.10 0.85 NConv [12] 829.98 233.26 2.60 1.03 S2D [21] 814.73 249.95 2.80 1.21 DepthNormal [32] 777.05 235.17 2.42 1.13 DeepLiDAR [25] 758.38 226.50 2.56 1.15 FuseNet [7] 752.88 221.19 2.34 1.14 CSPN++ [8] 743.69 209.28 2.07 0.90 ”
• PENet: Towards Precise and Efficient Image Guided Depth Completion (2021, methodology): “For instance, UberATG [20] applies continuous convolution on 3D points, ACMNet [21] exploits the graph propagation, DeepLiDAR [5] and PwP [17] use surface normal to introduce geometric constraints.”
• Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving (2019, methodology): “Here five methods are chosen for comparisons: Mono3D [3], Deep3DBox [23] and Multi-Fusion [37], ROI-10D [20], MonoGRNet [30] and Pseudo-LiDAR [31].”
• Learning Guided Convolutional Network for Depth Completion (2019, methodology): “2) Fusion Scheme of GuideNet and DepthNet: As described in subsection III-D, instead of using early or late feature fusion like existing methods [6], [7], [34], our approach fuses the decoder features of the GuideNet to the encoder features of the DepthNet.”