An End-to-End Transformer Model for 3D Object Detection

By Ishan Misra, Rohit Girdhar, Armand Joulin. IEEE International Conference on Computer Vision, 2021

623 citations recorded by the source corpus.

We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds. Compared to existing detection methods that employ a number of 3D-specific inductive biases, 3DETR requires minimal modifications to the vanilla Transformer block. Specifically, we find that a standard Transformer with non-parametric queries and Fourier positional embeddings is competitive with specialized architectures that employ libraries of 3D-specific operators with hand-tuned hyperparameters. Nevertheless, 3DETR is conceptually simple and easy to implement, enabling further improvements by incorporating 3D domain knowledge. Through extensive experiments, we show 3DETR outperforms the well-established and highly optimized VoteNet baselines on the challenging ScanNetV2 dataset by 9.5%. Furthermore, we show 3DETR is applicable to 3D tasks beyond detection, and can serve as a building block for future research.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/03a2befad038a9f29859295fdfcdbfa52c564622

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “The Transformer architecture [92] originally proposed for NLP tasks has been successfully applied in computer vision on images [11, 21, 70, 88, 93, 94], video [5, 8, 28, 29, 52, 66], and 3D data [60, 68, 103].”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “To generate instance queries, we rely on stacked Transformer decoders [5, 6] that iteratively attend to learned point features in a coarse-to-fine fashion using non-parametric queries [37].”
A Survey of Visual Transformers (2021, methodology): “Group-Free 3D DETR [118] applies a more specified and stronger structure than [117].”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, background): “Since then, numerous advancements have been made in point cloud based tasks such as 3D object classification [30, 32, 42, 53, 59, 66, 74], 3D object detection [30, 35, 40, 73] and 3D point cloud synthesis [1, 61].”
Revealing Occlusions with 4D Neural Fields (2022, background): “Recently, architectures that are built solely with self-attention as computational units have started to perform on par with or better than convolutional networks as generic feature extractors [38] in standard vision tasks such as object detection and segmentation [30, 46, 47] and point cloud–based ”
DETRs with Hybrid Matching (2022, methodology): “3DETR [46] introduced two important changes to the standard DETR [5], including non-parametric query embeddings and Fourier positional embeddings.”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “In these years, 3D vision has gained significant attention and development, driven by the rising popularity of autonomous driving [11, 69, 71], navigation [72, 76, 98], 3D scene understanding [2, 43, 46, 74], and robotics [31, 67].”
Embracing Single Stride 3D Object Detector with Sparse Transformer (2021, background): “Some of them focus on the indoor scene such as [17, 34, 63].”
Point2Seq: Detecting 3D Objects as Sequences (2022, methodology): “3DETR [21] utilizes sampled points as object queries.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, background): “, 2021), 3D object detection (Misra et al., 2021) and 3D reconstruction (Bozic et al.”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “For 3D understanding, transformer-based networks are also adopted for shape classiﬁcation, part segmentation [17, 64], 3D object detection from point clouds [32] and monocular images [62].”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43,53,79,89], etc .”
Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “[32] claims that position encoding is unnecessary for 3D Transformer-based networks because the xyz coordinates have already been used as the input features.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “The decoder layer follows the design of DETR [23] and the detailed architecture is provided in the supplementary Sec.”
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022, background): “Pre-aligned 3D visual understanding research [6, 12, 16, 17, 24, 25] is drawing significant attention in recent years due to the increasing demand of real-world applications such as augmented/virtual reality [1, 27, 31, 46], autonomous driving [23, 57] and robotics [2, 48].”