DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

By Yue Wang, Vitor Campanholo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, Justin M. Solomon. Conference on Robot Learning, 2021

1,126 citations recorded by the source corpus.

We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/48e2d76d35b44edc21d09d460021103ce997c804

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “SimMOD [41] employed a monocular 3D head for making proposals and a DETR3D [36] head for the final detection.”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, background): “The recent development of deep learning brings signiﬁcant breakthroughs in various perception tasks such as 3D object detection [22, 35, 42], object tracking [43, 56], and semantic segmentation [47, 57].”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “The pooled BEV features from different frames are directly concatenated and fed to following tasks. nuScenes val set We compare the proposed BEVDepth with other state-of-the-art methods like FCOS3D, DETR3D, BEVDet, PETR, BEVDet4D and BEVFormer on nuScenes val set.”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, background): “Given multi-frame and multi-view image input, VAD first encodes the image features with a backbone network and utilizes a group of BEV queries to project the image features to the BEV features [26,39,52].”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Instead of performing object detection in the perspective view, DETR3D [54] and PETR [27] design a DETR [68, 56]-based detection head with learnable object queries in the 3D space.”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, positive): “More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25].”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, methodology): “The high-performance conﬁguration dubbed BEVDet4D-Base scores high as 42.1% mAP and 54.5% NDS, which has surpassed all published results in vision-based 3D object detection [40, 41, 42, 22, 6, 18, 15].”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “Following prior works [28, 29, 32, 41], we use ResNet [8] or VoVNet [20] and FPN [16] for image feature extraction, it outputs multi-scale features for each image, denoted as for the k -th image.”
Revealing Occlusions with 4D Neural Fields (2022, background): “DETR3D [61] extends DETR [8] by computing the keys and values from multi-view images.”
A Survey of Visual Transformers (2021, background): “Camera-Based: MonoDTR [120], MonoDETR [121], DETR3D [122], TransFusion [123].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Other methods implicitly learn 3D features without producing explicit depth maps [53, 29, 55, 34, 52, 63].”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, neutral): “Given the set of sampled noisy trajectories { ˆ τ k } N infer k =1 from the anchored Gaussian distribution, we begin by applying deformable spatial cross-attention [29, 42, 62] to interact with Bird’s Eye View (BEV) or Perspective View (PV) features based on the trajectory coordinates.”
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Specifically, 3D perception [28, 30, 39, 45] is limited to detecting and tracking familiar objects, omitting rare objects and their unique attributes; motion prediction [16, 18, 33, 35, 56] and planning [2, 31, 38] focus on trajectory-level actions, often neglecting the decision-level interactions…”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Despite a concurrent work [5] aims to fuse multi-modalities information in a single network, its design is limited to one specific detection head [51] while our framework can be generalized to arbitrary architectures.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “The multi-camera 3D object detection methods can be divided into BEV-based [11, 10] and DETR-based [38, 21, 18] approaches.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “Another line of research performs 3D scene understanding experiments with only 2D ground truth supervi-sions [16, 27, 38, 47, 52, 55].”
VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, methodology): “Inspired by DETR [62] in 2D detection, DETR3D [63] links learnable 3D object queries with 2D images by camera projection matrices and enables end-to-end 3D bounding box prediction without non-maximum suppression (NMS).”