Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

By Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Murray Campbell, Kilian Q. Weinberger. 2018

1,180 citations recorded by the source corpus.

3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies --- a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations --- essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance --- raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, methodology): “orld [19]. Learning-based methods have shown promise in addressing these limitations by learning from data. One approach to improve upon SLAM methods is to directly estimate the geometry of the scene [20, 21, 22]. However, these methods are limited in that the geometry is only a partial description ”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “However, 3D localization from images is challenging [13, 12, 57, 80, 69, 66, 73].”
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection (2020, background): “[18] and [19] explore using stereo images to generate dense point cloud and conduct object detection using that cloud.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “Monocular Depth Estimation (MDE), which is a fundamental problem with broad applications in robotics [65], autonomous driving [63, 79], virtual reality [47], etc ., also requires a foundation model to estimate depth information from a single image.”
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection (2021, methodology): “Transform to 3D representations Another category of methods converts the input RGB image to other 3D representations, such as voxels [28] and point clouds [34].”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “PointNet architecture has been used in VoxelNet [45], Lidar-RCNN [14] to extract feature for a small region of irregular points.”
UniDepth: Universal Monocular Metric Depth Estimation (2024, background): “The precise pixel-wise depth estimation is crucial to understanding the geometric scene structure, with applications in 3D modeling [9], robotics [10, 52], and autonomous vehicles [31, 41].”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “In particular, pseudo point clouds in Figure 1a are transformed from image aided by predicted depth [1, 2], while the range-view image in Figure 1b is projected from point clouds [3, 4].”
Probabilistic and Geometric Depth: Detecting Objects in Perspective (2021, methodology): “Previous methods [10, 11] first use an extra cumbersome depth estimation model to complement 2D detectors on depth information.”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, methodology): “of [48] depends heavily on the accuracy of the estimated depth maps, while our method achieves accurate detection results when accurate depth maps are missing.”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, methodology): “Existing methods focus on deep representation learning [39] and geometry priors [29, 30, 46, 56].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Depth-based methods explicitly predict depth maps for image inputs to extract 3D geometric information of the scene before perception [54, 36, 44, 39, 28, 35, 30, 42, 21, 65].”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, positive): “Psuedo-lidar based approaches [24, 43, 44, 45] project to 3D points using a depth estimate and leverage 3D point based architectures (e.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “[147] and their successive work [148] propose to convert RGB images into pseudo-lidar representation by”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, background): “Methods transform image features into BEV features with the depth information from depth estimation [46] or categorical depth distribution [34].”
Delving into Localization Errors for Monocular 3D Object Detection (2021, methodology): “To better estimate the 3D bounding boxes, many methods are proposed for effectively using additional data [38, 26, 5, 39, 4, 12, 16, 13, 37, 25, 24, 3].”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, background): “Bird’s-eye-view (BEV) object detection has attracted more attention recently [17, 21, 25, 27, 29, 35, 42] due to its vast success in autonomous driving systems.”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, methodology): “An approach with recent empirical success is to separately train one network to do monocular depth prediction and another to do bird’s-eye-view detection separately [39] [40].”