Monte Carlo convolution for learning on non-uniformly sampled point clouds

By Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, À. Vinacua, Timo Ropinski. 2018

282 citations recorded by the source corpus.

Deep learning systems extensively use convolution operations to process input data. Though convolution is clearly defined for structured data such as 2D images or 3D volumes, this is not true for other data types such as sparse point clouds. Previous techniques have developed approximations to convolutions for restricted conditions. Unfortunately, their applicability is limited and cannot be used for general point clouds. We propose an efficient and effective method to learn convolutions for non-uniformly sampled point clouds, as they are obtained with modern acquisition techniques. Learning is enabled by four key novelties: first, representing the convolution kernel itself as a multilayer perceptron; second, phrasing convolution as a Monte Carlo integration problem, third, using this notion to combine information from multiple samplings at different levels; and fourth using Poisson disk sampling as a scalable means of hierarchical point cloud learning. The key idea across all these contributions is to guarantee adequate consideration of the underlying non-uniform sample distribution function from a Monte Carlo perspective. To make the proposed concepts applicable to real-world tasks, we furthermore propose an efficient implementation which significantly reduces the GPU memory required during the training process. By employing our method in hierarchical network architectures we can outperform most of the state-of-the-art networks on established point cloud segmentation, classification and normal estimation benchmarks. Furthermore, in contrast to most existing approaches, we also demonstrate the robustness of our method with respect to sampling variations, even when training with uniformly sampled data only. To support the direct application of these concepts, we provide a ready-to-use TensorFlow implementation of these layers at https://github.com/viscom-ulm/MCCNN.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, neutral): “, [32, 25, 20, 42, 44, 18, 2, 24, 33, 11] or from autoencoders [45].”
Flex-Convolution - Million-Scale Point-Cloud Learning Beyond Grid-Worlds (2018, background): “Recently, multiple attempts from the PointNet family [17,19,28] amongst others [14,25,10] proposed to handle irregular point clouds directly in a deep neural network.”
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (2019, background): “The last branch is 3D pseudo-continuous convolutional neural networks [12, 16].”
DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing (2019, unknown): “19 MC-Conv [11] 1k 0.”
PointConv: Deep Convolutional Networks on 3D Point Clouds (2018, methodology): “Also, we utilize density differently than [12], and we propose a deconvolution operator based on PointConv to perform semantic segmentation.”
ShellNet: Efficient Point Cloud Convolutional Neural Networks Using Concentric Shells Statistics (2019, background): “Beyond learning on unstructured point clouds, there have been some notable extension works, such as learning with hierarchical structures [28, 14, 35, 36], learning with selforganizing network [19], learning to map a 3D point cloud to a 2D grid [43, 8], addressing large-scale point cloud segmentatio”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, unknown): “Recent learning based approaches mainly include voxel-based [42; 46; 41; 23; 40; 11; 4] and point-based schemes [37; 19; 14; 16; 45].”
KPConv: Flexible and Deformable Convolution for Point Clouds (2019, methodology): “As shown by [41, 20, 13], the kernel of a point convolution can be implemented with a MLP, because of its ability to approximate any continuous function.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, unknown): “For point clouds (PCs) the pioneering PointNet based architectures [60, 61] spearheaded research, such as kernel point convolutions [72, 59, 33, 78], tree-based graph convolutions [67] normalizing flows [84], combinations of points and voxels for efficiency [46] or domain adaptation techniques [62].”
PCT: Point cloud transformer (2020, methodology): “[8] allows non-uniformly sampled point clouds; convolution is treated as a Monte Carlo integration problem.”
Differentiable surface splatting for point-based geometry processing (2019, methodology): “Hermosilla et al. [2019] propose an unsupervised point cloud cleaning method based on Monte Carlo convolution [Hermosilla et al. 2018].”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, background): “3D convolutional models [3, 17, 28, 31, 38, 50, 70, 85, 92] are widely used in 3D scene understanding [32, 82, 109, 116].”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “ints are assigned with the same weights when falling into the same grid. PointCNN [38] achieves permutation invariance through a ˜-conv transformation. Besides CNNs on discrete space, several methods [8,15,16,39,40,41,42,43] dene convolutional kernels on continuous space. Thomas et al. [15] propose ”
PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling (2020, methodology): “Methods [47, 44, 11, 22, 30, 40] directly use the relationship between neighborhoods and local centers to learn a dynamic weight for convolution, where ECC [30] and RS-CNN [22] use ad-hoc defined 6-D and 10-D vectors as edge relationship, PCCN [40] and PointConv [44] project the relative position of”
An End-to-End Transformer Model for 3D Object Detection (2021, background): “3D data can also be converted into a volumetric 3D grid by voxelization [1, 12, 15, 28, 35, 49, 56, 66].”
PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding (2018, background): “Recent advances of 3D semantic segmentation [30, 31, 46, 19, 35, 24, 9, 39, 40, 42, 33, 7, 26, 23] have accomplished promising performance in coarse-level segmentation on the ShapeNet Part dataset [3, 45].”
3D Point Capsule Networks (2018, methodology): “Due to this widespread use, recent works such as PointNet [26], PointNet++ [27], SO-Net [22], spherical convolutions [20], Monte Carlo convolutions [12] and dynamic graph networks [44] have all devised point cloud-specific architectures that exploited the sparsity and permutation-invariant propertie”
JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds (2019, background): “Some later works (Simonovsky and Komodakis 2017; Hermosilla et al. 2018; Xu et al. 2018) extract features of point clouds with learning continuous filters for convolution calculation.”