PointConv: Deep Convolutional Networks on 3D Point Clouds
By Wenxuan Wu, Zhongang Qi, Fuxin Li. 2018
1,894 citations recorded by the source corpus.
Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and the density functions through kernel density estimation. A novel reformulation is proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Geology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
In Computer Graphics and Computer-Aided Design
- Lukas Prantl
- Rohit Girdhar
- Ishan Misra
- Armand Joulin
- Erik J. Bekkers
- Vladlen Koltun
- Ji Hou
- Matthias Nießner
In Geology
Citations in the corpus
- Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “For PointConv, the input a cij = x ci − x cj and for LieConv the input a cij = Concat([log( v − 1 j u i ) , q i , q j ]) c .”
- Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, methodology): “Previous studies (Thomas et al. 2019; Wu, Qi, and Fuxin 2019) about point cloud semantic segmentation mainly focused on the complete or dense point cloud scenarios, which are post-processed by merging multiple collected Li-DAR or RGB-D sequences (e.g., ScanNet (Dai et al. 2017), For the sparse…”
- Point Transformer V3: Simpler, Faster, Stronger (2023, neutral): “However, in contrast to the progress made in 2D vision or NLP, the development of 3D backbones [16, 46, 62, 88] has been hindered in terms of scale, primarily due to the limited size and diversity of point cloud data available in separate domains [92].”
- Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation (2020, positive): “Mainstream methods [29, 36, 44, 41, 37, 24, 12, 50, 46, 38, 28, 30] of indoor point cloud segmentation learn the point features based on the raw point directly, which are often based on the pioneering work, i.e ., Point-Net, and promote the effectiveness of sampling, grouping and ordering to…”
- Fast end-to-end learning on protein surfaces (2020, methodology): “Numerous methods have been proposed to mimic surface operators with such convolution operators on meshes or point clouds – see Section 2 and especially [39, 25, 45, 40].”
- Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “We use these datasets and evaluate the performance of our methods on the indoor detection [12, 22, 64, 65, 114], scene segmentation [17, 68, 92, 101, 108], and outdoor detection tasks [15, 48, 76–78, 109, 111].”
- GemNet: Universal Directional Graph Neural Networks for Molecules (2021, background): “Moreover, since everything is based on efficient matrix products, this allows us to use the bilinear layer at practically no additional cost compared to a Hadamard product.”
- Review: deep learning on 3D point clouds (2020, methodology): “PointConv [37] Uniform sampling radius-nn MLP”
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (2021, methodology): “More explicit modeling of the spatial relations between points is performed with more recent methods (Li et al., 2018b; Liu et al., 2019c; Wu et al., 2019).”
- JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds (2019, methodology): “Recently, PointNet (Qi et al. 2017a) (Wu, Qi, and Fuxin 2019) proposes an inverse density scale to re-weight the continuous function learned by MLP and compensate the non-uniform sampling, while it also needs high GPU memory during training process.”
- PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, background): “Other approaches [51, 48, 57, 52, 21] enhance feature learning on local regions by dynamic context aggregation and attention modules.”
- RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, methodology): “Existing point sampling approaches [44, 33, 15, 12, 1, 60] can be roughly classified into heuristic and learningbased approaches.”
- Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “Abundant methods [7, 15, 16, 34, 35, 41, 52, 61, 62] have explored 3D point cloud segmentation and obtained decent performance.”
- PCT: Point cloud transformer (2020, background): “29] 85.2 84.0 83.4 86.7 77.8 90.6 74.7 91.2 87.5 82.8 95.7 66.3 94.9 81.1 63.5 74.5 82.6 P2Sequence [19] 85.2 82.6 81.8 87.5 77.3 90.8 77.1 91.1 86.9 83.9 95.7 70.8 94.6 79.3 58.1 75.2 82.8 PointConv [31] 85.7 - - - - - - - - - - - - - - - - PointCNN [17] 86.1 84.1 86.5 86.0 80.8 90.6 79.7 92.3 88.4”
- Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures (2021, methodology): “While we state our operation as an unstructured convolution as done in 3D point cloud learning (Hermosilla et al., 2018; Groh et al., 2018; Wu et al., 2019; Xiong et al., 2019; Thomas et al., 2019), it can also be understood in a message passing framework (Kipf & Welling, 2017) where hidden states…”
- PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies (2022, neutral): “[53, 22, 23] adaptively aggregate neighborhood features through weights determined by the local structure.”
- Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework (2022, methodology): “(Wu et al., 2019; Wang et al., 2019; Ran et al., 2021), to state-of-the-art Transformer-based models (Guo et al., 2021; Zhao et al., 2021).”
- Geometry-Informed Neural Operator for Large-Scale 3D PDEs (2023, background): “The similar idea has been explored in term of continuous 3D point cloud convolution [49–51].”