PETR: Position Embedding Transformation for Multi-View 3D Object Detection

By Yingfei Liu, Tiancai Wang, X. Zhang, Jian‐Yuan Sun. European Conference on Computer Vision, 2022

852 citations recorded by the source corpus.

In this paper, we develop position embedding transformation (PETR) for multi-view 3D object detection. PETR encodes the position information of 3D coordinates into image features, producing the 3D position-aware features. Object query can perceive the 3D position-aware features and perform end-to-end object detection. PETR achieves state-of-the-art performance (50.4% NDS and 44.1% mAP) on standard nuScenes dataset and ranks 1st place on the benchmark. It can serve as a simple yet strong baseline for future research. Code is available at \url{https://github.com/megvii-research/PETR}.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8bef2ad7d969f0e37d9c8b78d9c8d2997838f2cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “The pooled BEV features from different frames are directly concatenated and fed to following tasks. nuScenes val set We compare the proposed BEVDepth with other state-of-the-art methods like FCOS3D, DETR3D, BEVDet, PETR, BEVDet4D and BEVFormer on nuScenes val set.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “Meanwhile, other implicit manners in contemporary work [8, 9, 10] can hardly support explicit feature interactions in 3D space and restrict further knowledge transfer.”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, positive): “Specifically, we adopt the implementation by Liu et al. (2022), employing grid dimensions of ( T img , N cam , N patch , D ) .”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, neutral): “More recently, vision-based 3D object detection has gained more attention due to its low cost and rich semantic content [41, 46, 48, 22, 26, 33, 28, 13, 27, 15, 21, 31, 25].”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Instead of performing object detection in the perspective view, DETR3D [59], PETR [30] and Graph-DETR3D [11] design DETR [74, 61]-based detection heads with learnable object queries in the 3D space.”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Though lacking direct sensing of depth information, vision-based models empowered by surrounding cameras demonstrate promising performance on various 3D perception tasks such as depth estimation [18,45], semantic map reconstruction [1,20,51], and 3D object detection [28,31,49].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “Second, VAD utilizes a group of agent queries and map queries to learn the vectorized scene representation, including vectorized map and vectorized agent motion (Sec.”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “With the multiview camera images as input, various attempts for 2D-to3D transformation [40, 31, 20, 29] have been proposed for applications including 3D object detection [20, 31, 34], semantic map construction [40, 62, 44, 39], and motion pre-”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, methodology): “PETR [22] further develops the performance of this paradigm by introducing the 3D coordinate generation and position encoding.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “2D backbone (e.g., ResNet-50), and the 3D coordinates are generated from camera frustum space as described in PETR [24].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Other methods implicitly learn 3D features without producing explicit depth maps [53, 29, 55, 34, 52, 63].”
DETRs with Hybrid Matching (2022, methodology): “Inspired by the great success of DETR in object detection, many recent efforts have constructed various DETR-based approaches for segmentation tasks [8, 9, 25, 31, 76, 77] that aim for more accurate pixel localization and recognition, 3D object detection tasks [1, 20, 21, 30, 34, 39, 39, 40, 44, 45,”
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, methodology): “Pioneer’s works [43, 27, 15, 25, 17, 44] focus on the efficient transformation from multiple perspective views to a unified 3D space in a single frame.”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “Bird’s-eye-view (BEV) object detection has attracted more attention recently [17, 21, 25, 27, 29, 35, 42] due to its vast success in autonomous driving systems.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “Furthermore, Group DETR outperforms baseline methods for multi-view 3D object detection [37, 38] and instance segmentation [9].”