Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving

By Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Murray Campbell, Kilian Q. Weinberger. International Conference on Learning Representations, 2019

446 citations recorded by the source corpus.

Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap. In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation. Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects --- currently the primary weakness of pseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to de-bias our depth estimation. We propose a depth-propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection --- outperforming the previous state-of-the-art detection accuracy for faraway objects by 40%. Our code is available at this https URL.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6f1ecfb639b9e89460c4d1f71ced86f9a9b86ee2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, methodology): “Leveraging exciting recent progress in depth estimation [11, 13, 15, 16, 31], pseudo-lidar detectors [69, 40, 57] first use a pre-trained depth network to compute an intermediate pointcloud representation, which is then fed to a 3D detection network.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “It often involves projecting geometric features from various perception sensors onto image space, such as bird’s-eye view [28], [38] [22], [36], [45] and range view [39], [46], [47].”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “In particular, pseudo point clouds in Figure 1a are transformed from image aided by predicted depth [1, 2], while the range-view image in Figure 1b is projected from point clouds [3, 4].”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “One typical approach is transforming image-based depth maps to pseudo-LiDAR to mimic the LiDAR signal [12, 13, 14].”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “, 2019), leveraging depth images (Xu & Chen, 2018) or pseudo-LiDAR point clouds (Wang et al., 2019b; You et al., 2020), introducing geometry constraints (Mousavian et al.”
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection (2021, background): “Recent work [36, 23, 33, 24] has made great progress following this approach and shown promising performance.”
PointPainting: Sequential Fusion for 3D Object Detection (2019, background): “Indeed, despite recent fusion research [18, 27, 13, 33], the top methods on Figure 1.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “Monocular Depth Estimation (MDE), which is a fundamental problem with broad applications in robotics [65], autonomous driving [63, 79], virtual reality [47], etc ., also requires a foundation model to estimate depth information from a single image.”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, methodology): “, pseudo-LiDAR [52], [54], [55], where the LiDARbased detection models can operate on pseudo-LiDAR for”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “object detection [22, 12, 66, 7, 44, 31, 34, 61, 33, 37] and motion forecasting [36, 5, 35, 63, 6, 19, 38].”
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection (2020, background): “[18] and [19] explore using stereo images to generate dense point cloud and conduct object detection using that cloud.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”
HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation (2020, background): “Especially in the depth estimation task, pixel level disparity is more important since it is inversely proportional to the square of depth error(You et al. 2019).”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “[147] and their successive work [148] propose to convert RGB images into pseudo-lidar representation by”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, methodology): “An approach with recent empirical success is to separately train one network to do monocular depth prediction and another to do bird’s-eye-view detection separately [39] [40].”