BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

By Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, Song Han. IEEE International Conference on Robotics and Automation, 2022

1,613 citations recorded by the source corpus.

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we propose BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift the key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than $\mathbf{40}\times$. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on the nuScenes benchmark, achieving 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9× lower computation cost. Code to reproduce our results is available at https://github.com/mit-han-lab/bevfusion.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Instrumentation

View Paper: https://www.semanticscholar.org/paper/4ac46b5bbd5728b51279d07c1730b509943a982e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

DETRs with Hybrid Matching (2022, neutral): “How to apply DETR for 3 D object detection has attracted a lot of efforts [46, 20, 19, 1, 38, 31, 36].”
NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, methodology): “With the image featmap M and LiDAR featmap M , we can aggregate them to obtain multi-modal featmap M ∈ R 𝐻 × 𝑊 × 𝑑 𝑚 as many detection works [20, 28].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, neutral): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”
VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (2023, methodology): “This dense centerbased prediction has been adopted by several dense-head approaches [29, 32].”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, neutral): “More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, neutral): “Formalizing this task for the end-to-end driving domain and incorporating techniques from the zero-shot/few-shot learning literature are the key steps toward achieving this [269], [270].”
MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (2022, methodology): “And for fair comparison with IPM and LSS, GKT and Deformable Attention both adopt one-layer conﬁguration.”
Planning-oriented Autonomous Driving (2022, methodology): “However, such a scheme may cause undesirable “negative transfer” [23,64].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “camera images as inputs, it has demonstrated competitive performance in various 3D perception tasks including depth estimation [18, 57], 3D object detection [29, 28, 35, 33, 21], and semantic map construction [65, 20, 1].”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “In addition, concurrent works [27, 58] also address this problem and propose effective LiDAR-camera 3D perception models.”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, methodology): “As a popular alternative, BEV [22,27,28,32] models the 3D scene with a 2D feature map B ∈ H × W × C which encodes the top view of the scene.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We compare our method with the latest SOTA approaches trained on the KITTI360, including NeRF-based PNF [6] and 3D Gaussian-based 3DGS [4].”