3D Object Detection with Pointformer

By Xuran Pan, Zhuofan Xia, Shiji Song, L. Li, Gao Huang. Computer Vision and Pattern Recognition, 2020

460 citations recorded by the source corpus.

Feature learning for 3D object detection from point clouds is very challenging due to the irregularity of 3D point cloud data. In this paper, we propose Pointformer, a Transformer backbone designed for 3D point clouds to learn features effectively. Specifically, a Local Transformer module is employed to model interactions among points in a local region, which learns context-dependent region features at an object level. A Global Transformer is designed to learn context-aware representations at the scene level. To further capture the dependencies among multi-scale representations, we propose Local-Global Transformer to integrate local features with global features from higher resolution. In addition, we introduce an efficient coordinate refinement module to shift down-sampled points closer to object centroids, which improves object proposal generation. We use Pointformer as the backbone for state-of-the-art object detection models and demonstrate significant improvements over original models on both indoor and outdoor datasets.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/e374193760d53abd55be0846665dde9e028cc42b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “The Transformer architecture [92] originally proposed for NLP tasks has been successfully applied in computer vision on images [11, 21, 70, 88, 93, 94], video [5, 8, 28, 29, 52, 66], and 3D data [60, 68, 103].”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “Given an input image, AdaViT learns to adaptively derive policies on which patches , self-attention heads and transformer blocks to use or activate in the transformer backbone conditioned on the input image, encouraging us-ing less computation while maintaining the classiﬁcation accuracy.”
Embracing Single Stride 3D Object Detector with Sparse Transformer (2021, background): “For autonomous driving scenarios, Pointformer [35] proposes a point-based local and global attention module directly operating on point clouds.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “Concurrent work [37, 91] also uses the Transformer architecture for 3D.”
SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (2021, background): “Then, the success of transformerbased 2D CV studies have drawn the attention of 3D point cloud research, where pioneering studies like PointTransformer [60], PCT [9] and Pointformer [38] have introduced such framework in the encoding process of point cloud to learn the representation.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “Despite the popularity of adopting the transformer architecture as a detection head in 2D [2], 3D detection models for outdoor scenarios mostly utilize the transformer for feature extraction [21, 24, 35].”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “domain though and recent Transformer-based methods focus on either 3D object detection [34, 37, 39] or 3D semantic segmentation [29, 40, 63].”
On the Integration of Self-Attention and Convolution (2021, methodology): “Furthermore, transformer paradigm is adopted in detection [2,7,56], segmentation [45,52,54], point cloud recognition [17, 32] and other vision tasks [8, 34].”
A Survey on Vision Transformer (2020, methodology): “Transformer-based object detection methods are broadly categorized into two groups: transformer-based set prediction methods [19], [291], [210], [284], [154] and transformer-based backbone methods [10], [166], as shown in Fig.”
Voxel Transformer for 3D Object Detection (2021, background): “Point-based approaches [26, 19, 34, 35] directly operate and generate 3D bounding boxes on point clouds.”
Improving 3D Object Detection with Channel-wise Transformer (2021, background): “An important family of models is PointNet [22] and its variants [23, 19, 25], which use a flexible receptive field to aggregate features by local regions and permutation-invariant network.”
A Survey of Visual Transformers (2021, background): “Fine-Grained: Pointformer [110], SST [111],”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “This foreground point segmentation strategy has been adopted in most point-based detectors (Shi et al., 2019; Yang et al., 2020c, 2018c; Pan et al., 2021), with improvements such as adding centerness scores (Yang et al.”
Vision Transformer with Deformable Attention (2022, unknown): “Stage 1 ( 56 × 56 ) former [29].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Then, the following works adopt improved designs on various vision tasks, including representation learning [23], [24], object detection [25], segmentation [26], low-level image processing [27], video understanding [28], 3D scene understanding [29], and image/video generation [30].”