BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

By Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, T. Tang, Bing Wang, Zhi Tang. Neural Information Processing Systems, 2022

712 citations recorded by the source corpus.

Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion, whose camera stream does not depend on the input of LiDAR data, thus addressing the downside of previous methods. We empirically show that our framework surpasses the state-of-the-art methods under the normal training settings. Under the robustness training settings that simulate various LiDAR malfunctions, our framework significantly surpasses the state-of-the-art methods by 15.7% to 28.9% mAP. To the best of our knowledge, we are the first to handle realistic LiDAR malfunction and can be deployed to realistic scenarios without any post-processing procedure. The code is available at https://github.com/ADLab-AutoDrive/BEVFusion.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4e7487374ba0807f233ebe930d2f7ac507e8830a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Hongsheng Li
2. Shaoshuai Shi
3. Xiaogang Wang
4. Jiaya Jia
5. Jiageng Mao
6. Xiaojuan Qi
7. Xizhou Zhu
8. Yunpeng Zhang

In Artificial Intelligence

1. Hongsheng Li
2. Yu Qiao
3. Xiaogang Wang
4. Jifeng Dai
5. Jiaya Jia
6. Jie Zhou
7. Jiwen Lu
8. Andreas Geiger

In Computer Vision and Pattern Recognition

1. Xiaogang Wang
2. Luc Van Gool
3. Dacheng Tao
4. Hongsheng Li
5. Ming-Hsuan Yang
6. Yu Qiao
7. Chen Change Loy
8. Yi Yang

Citations in the corpus

• End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “Inspired by the progress in perception, it is beneficial to model modalities in a unified space such as BEV [131], [132].”
• DETRs with Hybrid Matching (2022, neutral): “How to apply DETR for 3 D object detection has attracted a lot of efforts [46, 20, 19, 1, 38, 31, 36].”
• SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Depth-based methods explicitly predict depth maps for image inputs to extract 3D geometric information of the scene before perception [54, 36, 44, 39, 28, 35, 30, 42, 21, 65].”
• Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, methodology): “As the information in outdoor scenes is not isotropically distributed, modern methods collapse the height dimension and mainly focus on the ground plane (bird’s-eye-view) where information varies the most [21,27,29,32,38,49,51].”
• DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We compare our method with the latest SOTA approaches trained on the KITTI360, including NeRF-based PNF [6] and 3D Gaussian-based 3DGS [4].”
• Planning-oriented Autonomous Driving (2022, background): “Note that UniAD is not confined to a specific BEV encoder, and one can utilize other alternatives to extract richer BEV representations with long-term temporal fusion [31, 74] or multi-modality fusion [58,64].”
• VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (2023, methodology): “This dense center-based prediction has been adopted by several dense-head approaches [30, 33].”
• 3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “From the results we can observe that fusion at any place can boost detection performance over single-modality baselines, and fusion in the BEV space (Liu et al., 2022b; Liang et al., 2022; Bai et al., 2022) is more popular recently for its performance and efficiency.”