BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

By Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, Jifeng Dai. European Conference on Computer Vision, 2022

1,985 citations recorded by the source corpus.

3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design spatial cross-attention that each BEV query extracts the spatial features from the regions of interest across camera views. For temporal information, we propose temporal self-attention to recurrently fuse the history BEV information. Our approach achieves the new state-of-the-art 56.9\% in terms of NDS metric on the nuScenes \texttt{test} set, which is 9.0 points higher than previous best arts and on par with the performance of LiDAR-based baselines. We further show that BEVFormer remarkably improves the accuracy of velocity estimation and recall of objects under low visibility conditions. The code is available at \url{https://github.com/zhiqi-li/BEVFormer}.

View Paper: https://www.semanticscholar.org/paper/a824c6e214dd0118f70af8bb05d67d94a858d076

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “For driving-speciﬁc representations, researchers introduce the concept of bird’s-eye-view (BEV), fusing different sensor modalities and temporal information within a uniﬁed 3D space [131], [132], [179], [180].”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Considering the dimension gap between the 2D image input and the 3D prediction, recent studies for vision-based 3D perception first construct the BEV feature representations and then perform various downstream tasks on the BEV space [19, 28, 30, 37, 58, 38, 60, 40, 18, 1, 42].”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Particularly, we utilize image-based 3D object detector [37] and online HDMap predictor [38] to obtain the perception re-sults on the generated videos.”
DETRs with Hybrid Matching (2022, background): “The follow-up BEVFormer [28] further extended DETR 3 D via exploiting spatial and temporal information and interacting with spatial and temporal space with a set of predeﬁned grid-shaped BEV queries.”
VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, methodology): “BEVFormer [65] proposes a spatiotemporal transformer that aggregates BEV features from current and previous features via deformable attention [66].”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, unknown): “End-to-end autonomous driving has gained signiﬁcant attention in recent years due to advancements in perception models (detection [5, 17, 24, 42], tracking [54–56], online mapping [ 27, 28, 32], etc .)”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Given that the detectors [82, 84] take image sequences as inputs, Drive-WM has to accomplish all steps of the denoising process before perception.”
MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (2022, methodology): “And for fair comparison with IPM and LSS, GKT and Deformable Attention both adopt one-layer conﬁguration.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, unknown): “Both UniAD and PARA-Drive use a BEVFormer backbone [31], which encodes the eight surround-7 Implementation details for all methods are provided in the supplementary material.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “This idea has also been adopted by several concurrent papers on BEV semantic segmentation from image inputs [81], [82], [83], [84], [85].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “3.4).”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, background): “We report the results for two additional problems: multi-view 3D object detection [24, 32, 37, 38] and instance segmentation [9, 30], to further demonstrate the effectiveness.”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, methodology): “It also exceeds the concurrent work of BEVFormer [18] by +3.4% NDS and signiﬁcantly ex-6 Table 3: Results of ablation study on the nuScenes val set.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, background): “[30, 10] aquires BEV features through spatial cross-attention with pre-defined BEV queries.”
Planning-oriented Autonomous Driving (2022, methodology): “Following BEVFormer [55], the Hungarian loss is adopted for each paired result, which is a linear combination of a Focal loss [61] for class labels and an l 1 for 3D boxes localization.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “Then multi-task heads are used for 3D detection and segmentation.”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, background): “LMDrive takes as input the language instruction and multi-modal multi-view sensor data, and outputs control signals in real time to drive in complex scenarios. together [2, 26, 41, 57]; and 2) an end-to-end approach that directly converts sensor data to control signals via a neural network [18, 39].”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Specifically, 3D perception [28, 30, 39, 45] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [16, 18, 33, 35, 56] and planning [2, 31, 38] focus on trajectory-level actions, often neglecting the decision-level interactions…”