Multimodal Virtual Point 3D Detection

By Tianwei Yin, Xingyi Zhou, Philipp Krahenbuhl. Neural Information Processing Systems, 2021

366 citations recorded by the source corpus.

Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution and cost. For autonomous driving, this means that large objects close to the sensors are easily visible, but far-away or small objects comprise only one measurement or two. This is an issue, especially when these objects turn out to be driving hazards. On the other hand, these same objects are clearly visible in onboard RGB sensors. In this work, we present an approach to seamlessly fuse RGB sensors into Lidar-based 3D recognition. Our approach takes a set of 2D detections to generate dense 3D virtual points to augment an otherwise sparse 3D point cloud. These virtual points naturally integrate into any standard Lidar-based 3D detectors along with regular Lidar measurements. The resulting multi-modal detector is simple and effective. Experimental results on the large-scale nuScenes dataset show that our framework improves a strong CenterPoint baseline by a significant 6.6 mAP, and outperforms competing fusion approaches. Code and more visualizations are available at https://tianweiy.github.io/mvp/

View Paper: https://www.semanticscholar.org/paper/9eaf5cc6f632b456e1d2959a93965c91f5c91fae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhijian Liu
2. Xiaojuan Qi
3. Jiaya Jia
4. Song Han
5. Yanwei Li
6. Liang-Chieh Chen
7. Kaiming He
8. Ji Lin

Citations in the corpus

• BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Previous fusion methods can be broadly categorized into (a) point-level fusion mechanism [43, 46, 47, 46, 17, 60] that project image features onto raw point clouds, and (b) feature-level fusion mechanism [4, 61, 1, 21] that projects LiDAR feature or proposals on each view image separately to…”
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “We refer the readers to previous 3D detection papers [1, 67, 68] for more details.”
• Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “Specifically, point-level fusion [30, 31, 32, 7] combines features from different modalities at the early stage of the network, which enables sufficient interaction.”
• FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, background): “MVP [42] generate dense virtual points with semantic label to make more use of image information.”