DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

By Yingwei Li, Adams Wei Yu, Tianjian Meng, Benjamin Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo-Xun Wu, Yifeng Lu, Denny Zhou and 3 more. Computer Vision and Pattern Recognition, 2022

531 citations recorded by the source corpus.

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods [34], [36] simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and LearnableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves Point-Pillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7,8.9, and 6.2 LEVEL_2 APH, respectively. Notably, our models achieve state-of-the-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/5ffca96f4becdab649f085699594caa7c5c03e86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022, background): “3D visual understanding research [6, 12, 16, 17, 24, 25] is drawing significant attention in recent years due to the increasing demand of real-world applications such as augmented/virtual reality [1, 27, 31, 46], autonomous driving [23, 57] and robotics [2, 48].”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, methodology): “One is the Test-Time Augmentation (TTA), which is of great help to the 3D object detection task with LiDAR [6, 36].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Advanced Transformer-based fusion mechanisms in the perception ﬁeld, such as [149], [150], hold promise for application to the end-to-end driving task.”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, unknown): “, first detect the object in 2D images, then use the obtained information to further process point clouds; more recent works [21,28] develop end-to-end fusion pipelines and leverage cross-attention [38] to perform feature alignment.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “BEVFusion outperforms the previous state-of-the-art multi-modal detector, DeepFusion [3] with 60% of input frames.”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “Similarly, DeepFusion [21] projects LiDAR features on each view image as queries and then leverages cross-attention for two modalities.”