Is Pseudo-Lidar needed for Monocular 3D Object detection?

By Dennis Park, Rares Ambrus, Vitor Campanholo Guizilini, Jie Li, Adrien Gaidon. IEEE International Conference on Computer Vision, 2021

411 citations recorded by the source corpus.

Recent progress in 3D object detection from single images leverages monocular depth estimation as a way to produce 3D pointclouds, turning cameras into pseudo-lidar sensors. These two-stage detectors improve with the accuracy of the intermediate depth estimation network, which can itself be improved without manual labels via large-scale self-supervised learning. However, they tend to suffer from overfitting more than end-to-end methods, are more complex, and the gap with similar lidar-based detectors remains significant. In this work, we propose an end-to-end, single stage, monocular 3D object detector, DD3D, that can benefit from depth pre-training like pseudo-lidar methods, but without their limitations. Our architecture is designed for effective information transfer between depth estimation and 3D detection, allowing us to scale with the amount of unlabeled pre-training data. Our method achieves state-of-the-art results on two challenging benchmarks, with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark, and 41.5% mAP on NuScenes.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Instrumentation

View Paper: https://www.semanticscholar.org/paper/8620b942d3e31261971ce7b5d50f652a09e0aa62

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Instrumentation

Citations in the corpus

BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, methodology): “ceeds those relied on additional data for pre-training like DD3D [29], DETR3D [42], and PETR [22].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “To instantiate the set of object tracks in a scene, we first run an RGB-only 3D object detector [41] and tracker [57].”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “Previous works [6,33,49,34,48] mainly solved this problem from the perspective of monocular object detection.”
UniDepth: Universal Monocular Metric Depth Estimation (2024, background): “The precise pixel-wise depth estimation is crucial to understanding the geometric scene structure, with applications in 3D modeling [9], robotics [10, 52], and autonomous vehicles [31, 41].”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “More importantly, when using DD3D [20] depth pretrained models for our camera backbone, 4-beam LiDAR with cameras outperform 32beam LiDAR and achieve 56.”
BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View (2021, background): “It is the up-todate popular benchmark for 3D object detection [42, 44, 43, 29] and BEV semantic segmentation [34, 30, 28, 47].”
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries (2021, methodology): “On the test set (Table 2), our method outperforms all existing methods as of 10/13/2021; our method uses the same backbone as DD3D [37] for a fair comparison.”
3D Object Detection for Autonomous Driving: A Survey (2021, methodology): “+ Mono3D [44], CVPR′16 + DD3D [20], ICCV′21”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “† indicates that V2-99 [13] was pretrained on the depth estimation task with extra data [27].”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “Note that, people discover that bird’s eye view (BEV) has been an de-facto standard for autonomous driving scenarios as, generally speaking, car cannot ﬂy [20, 23, 39, 16, 54, 33].”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “The backbone is pretrained on DDAM15M dataset [28] and train set of Nuscenes [2].”