PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
By Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas. 2016
18,024 citations recorded by the source corpus.
Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.
Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition, Geology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Andreas Geiger
- Vladlen Koltun
- Jitendra Malik
- Marc Pollefeys
- Leonidas J. Guibas
- Xiaogang Wang
- Matthias Nießner
- Hongsheng Li
In Computational Mechanics
- Andreas Geiger
- Leonidas J. Guibas
- Matthias Nießner
- Michael Niemeyer
- Gerard Pons-Moll
- Charles Ruizhongtai Qi
- Or Litany
- Thomas A. Funkhouser
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Trevor Darrell
- Andrea Vedaldi
- Andrew Zisserman
In Geology
- Leonidas J. Guibas
- Charles Ruizhongtai Qi
- Matthias Nießner
- Or Litany
- Binh-Son Hua
- Sai-Kit Yeung
- Vladlen Koltun
- Ishan Misra
Citations in the corpus
- Inductive Representation Learning on Large Graphs (2017, neutral): “All models use rectified linear units as a non-linear activation function.”
- VAIN: Attentional Multi-agent Predictive Modeling (2017, neutral): “Another recent approach is PointNet [18] where every point in a point cloud is embedded by a deep neural net, and all embeddings are pooled globally.”
- Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “We use permutation-invariant instance-pooling operations to process these sets similarly to Qi et al. (2017) and as formalized in Zaheer et al. (2017).”
- 3D Instance Segmentation of MVS Buildings (2021, methodology): “Volumetric-based methods [3], [4] and point cloud-based methods [5], [6] are limited by memory and computing power, thus are mainly used for the segmentation of relatively small indoor scenes [7]–[12].”
- PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017, methodology): “Note that PointNet (vanilla) in Table 2 is the the version in [20] that does not use transformation networks, which is equivalent to our hierarchical net with only one level.”
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection (2017, background): “Recently, Qi et al.[31] proposed PointNet, an end-toend deep neural network that learns point-wise features directly from point clouds.”
- Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “For point clouds we use the PointNet encoder [54].”
- Relational inductive biases, deep learning, and graph networks (2018, methodology): “Relation Networks (Raposo et al., 2017; Santoro et al., 2017) bypass the node update entirely and predict the global output from pooled edge information directly (see also Figure 4e), Deep Sets (Zaheer et al., 2017) bypass the edges update completely and predict the global output from pooled nodes i”
- Flexible Neural Representation for Physics Prediction (2018, background): “Representations using 3D convolution or point clouds are better at maintaining object shape [46, 47, 10, 36, 37], but do not entirely capture object permanence, and can be computationally inefficient.”
- How Powerful are Graph Neural Networks? (2018, background): “Qi et al. (2017) empirically show that the max-pooling aggregator learns to identify the skeleton of a 3D point cloud and that it is robust to noise and outliers.”
- SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, background): “DeepSet[30] 3DmFV[3] Set Transformer[12] PointNet[15] SpiderCNN[29] Tensor Field +z[20] PointNet++[16] SE(3)-Transf.”
- Set2Graph: Learning Graphs From Sets (2020, methodology): “…paradigm for designing deep models was used for many different tasks, data modalities and transformations, e.g., set learning (Zaheer et al., 2017; Qi et al., 2017), graph learning (Kipf & Welling, 2016; Gilmer et al., 2017; Veliˇckovi´c et al., 2017; Xu et al., 2019; Kondor et al., 2018; Maron…”
- Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “While it seems natural to extend the convolution operations from 2D to 3D, it is shown that performing convolutions on a point cloud is not a trivial task [30, 49].”
- Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds (2018, methodology): “The networks presented in Qi et al. [11, 12] operate on point clouds and use symmetric functions to encode permutation invariance, but these networks do not include rotation equivariance.”
- Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “4 PointNet [Qi et al. 2017b] 86.”