BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection

By Junjie Huang, Guan Huang. arXiv.org, 2022

526 citations recorded by the source corpus.

Single frame data contains finite information which limits the performance of the existing vision-based multi-camera 3D object detection paradigms. For fundamentally pushing the performance boundary in this area, a novel paradigm dubbed BEVDet4D is proposed to lift the scalable BEVDet paradigm from the spatial-only 3D space to the spatial-temporal 4D space. We upgrade the naive BEVDet framework with a few modifications just for fusing the feature from the previous frame with the corresponding one in the current frame. In this way, with negligible additional computing budget, we enable BEVDet4D to access the temporal cues by querying and comparing the two candidate features. Beyond this, we simplify the task of velocity prediction by removing the factors of ego-motion and time in the learning target. As a result, BEVDet4D with robust generalization performance reduces the velocity error by up to -62.9%. This makes the vision-based methods, for the first time, become comparable with those relied on LiDAR or radar in this aspect. On challenge benchmark nuScenes, we report a new record of 54.5% NDS with the high-performance configuration dubbed BEVDet4D-Base, which surpasses the previous leading method BEVDet-Base by +7.3% NDS. The source code is publicly available for further research at https://github.com/HuangJunJie2017/BEVDet .

Subfields: Computer Vision and Pattern Recognition, Instrumentation

View Paper: https://www.semanticscholar.org/paper/f28d933ed02494fe04aa462c5aeb8b06b7f0e8f3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Instrumentation

Citations in the corpus

Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, background): “This paradigm requires a large receptive field to alleviate this problem [16, 39, 27].”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “BEVDet [106, 105] improves LSS [219] with a four-step multi-view detection pipeline, where the image view encoder encodes features from multi-view images, the view transformer transforms image features from camera views to the bird’s-eye view, the BEV encoder further encodes the BEV features, and th”
DETRs with Hybrid Matching (2022, background): “How to apply DETR for 3D object detection has attracted a lot of efforts [46, 20, 19, 1, 38, 31, 36].”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “LSS [3] proposes a view transform method that explicitly predicts depth distribution and projects image features onto bird’s-eye view (BEV), which has been proved practical for 3D object detection [2, 1, 17].”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “A similar idea can also be found in BEVDet [15, 14], the state-of-the-art method in multi-view 3D object detection.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “For example, BEVDet4D [11] explicitly aligns the BEV feature of previous frame with current frame by pose transformation.”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, background): “Their follow-up works (Huang et al., 2021; Huang and Huang, 2022; Li et al., 2022a; Xie et al., 2022) learn a depth-based implicit projection to project image features to BEV space.”