Modeling Point Clouds With Self-Attention and Gumbel Subset Sampling

By Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, Qi Tian. 2019

441 citations recorded by the source corpus.

Geometric deep learning is increasingly important thanks to the popularity of 3D sensors. Inspired by the recent advances in NLP domain, the self-attention transformer is introduced to consume the point clouds. We develop Point Attention Transformers (PATs), using a parameter-efficient Group Shuffle Attention (GSA) to replace the costly Multi-Head Attention. We demonstrate its ability to process size-varying inputs, and prove its permutation equivariance. Besides, prior work uses heuristics dependence on the input data (e.g., Furthest Point Sampling) to hierarchically select subsets of input points. Thereby, we for the first time propose an end-to-end learnable and task-agnostic sampling operation, named Gumbel Subset Sampling (GSS), to select a representative subset of input points. Equipped with Gumbel-Softmax, it produces a "soft" continuous subset in training phase, and a "hard" discrete subset in test phase. By selecting representative subsets in a hierarchical fashion, the networks learn a stronger representation of the input sets with lower computation cost. Experiments on classification and segmentation benchmarks show the effectiveness and efficiency of our methods. Furthermore, we propose a novel application, to process event camera stream as point clouds, and achieve a state-of-the-art performance on DVS128 Gesture Dataset.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, background): “Motivated by a successes across a wide range of tasks in deep learning such as language modeling [31], image recognition [18], graph-based problems [32], and relational reasoning [30, 9], a recent stream of work has applied forms of self-attention algorithms to point cloud data [44, 42, 16].”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Self-attention has several properties that make it a good fit for vision: (a) content-based interactions as opposed to content-independent interactions of convolution; (b) parameter-independent scaling of receptive field size as opposed to parameter-dependent scaling of convolution; (c) empirical ab”
Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene (2020, background): “Categories of pointbased deep learning models mainly include feature pooling models [15, 23, 24, 43], convolution-based models [20, 30, 34], graph-based models [18, 33, 37] and attention-based models [41, 42]. Although most of these models are proposed for a single task, they can be involved in mult”
Point Transformer (2020, methodology): “[36] introduce a network architecture that replaces traditional subsampling methods like furthest point sampling (FPS) with an attention-based selection process using the gumbel-softmax function, which is similar to the proposed SortNet module.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “Point-wise MLP based architectures [17, 83] such as PointNet [44] and PointNet++ [45] use permutation equivariant set aggregation (downsam-pling) and pointwise MLPs to learn effective representations.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Transformers have also recently gained increasing attention in point cloud-based HAR [289], [294], [296], [297].”
Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis (2021, methodology): “By using gumbel-softmax [9, 23, 45] 1 as the scoring function, logits can be converted into an one-hot vector based on the arg max index.”
Attention mechanisms in computer vision: A survey (2021, unknown): “[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, methodology): “• Continuous Relaxation based Sampling (CRS): CRS approaches [1, 66] use the reparameterization trick to relax the sampling operation to a continuous domain for endto-end training.”
Review: deep learning on 3D point clouds (2020, methodology): “Other sampling methods include uniform sampling and Gumbel Subset Sampling [28].”
3D Object Detection with Pointformer (2020, background): “In the field of 3D vision, PAT [38] designs novel group shuffle attentions to capture long range dependencies in point clouds.”
PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling (2020, methodology): “Among these sampling algorithms, farthest point sampling (FPS) [25], Poisson disk sampling (PDS) [11], and Gumbel subset sampling (GSS) [48] are proposed in previous works.”