SPLATNet: Sparse Lattice Networks for Point Cloud Processing

By Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz. 2018

793 citations recorded by the source corpus.

We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice. Naïvely applying convolutions on this lattice scales poorly, both in terms of memory and computational cost, as the size of the lattice increases. Instead, our network uses sparse bilateral convolutional layers as building blocks. These layers maintain efficiency by using indexing structures to apply convolutions only on occupied parts of the lattice, and allow flexible specifications of the lattice structure enabling hierarchical and spatially-aware feature learning, as well as joint 2D-3D reasoning. Both point-based and image-based representations can be easily incorporated in a network with such layers and the resulting model can be trained in an end-to-end manner. We present results on 3D segmentation tasks where our approach outperforms existing state-of-the-art techniques.

Subfields: Computational Mechanics, Environmental Engineering, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Leonidas J. Guibas
2. Or Litany
3. Charles Ruizhongtai Qi
4. Li Jiang
5. Rohit Girdhar
6. Ishan Misra
7. Armand Joulin
8. Yue Wang

In Environmental Engineering

1. Cyrill Stachniss
2. Jens Behley
3. Jitendra Malik
4. Christopher Bongsoo Choy
5. Ignacio Vizzo
6. Charles Ruizhongtai Qi
7. Leonidas J. Guibas
8. Silvio Savarese

In Geology

1. Leonidas J. Guibas
2. Charles Ruizhongtai Qi
3. Or Litany
4. Li Jiang
5. Ishan Misra
6. Rohit Girdhar
7. Jiwen Lu
8. Armand Joulin

Citations in the corpus

• Dynamic Graph CNN for Learning on Point Clouds (2018, background): “2017], sparse network lattice [Su et al. 2018], or spline [Fey et al.”
• PointCNN: Convolution On X-Transformed Points (2018, background): “[43, 3, 44] propose to first “interpolate” or “project” features into predefined regular domains, where typical CNNs can be applied.”
• Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “In [110], bilateral Conv layers (BCL) are applied to perform convolutions on occupied parts of the lattice for hierarchical and spatially-aware feature learning.”
• Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, neutral): “Recent learning based approaches mainly include voxel-based [42; 46; 41; 23; 40; 11; 4] and point-based schemes [37; 19; 14; 16; 45].”
• PointConv: Deep Convolutional Networks on 3D Point Clouds (2018, methodology): “Since our PointConv only takes 3D point clouds as input, for fair comparison, we only compare our result with the SPLATNet3D in [35].”
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020, background): “Second, is the choice of neighborhood aggregation mechanism which could either be hierarchical [48, 33, 82, 16, 35], spatial CNN-like [30, 73, 37, 85, 59], spectral [78, 62, 65] or graph-based [72, 64, 68, 54].”
• SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, methodology): “To overcome the limitations of the voxel-based representation, such as the exploding memory consumption when the resolution of the voxel grid increases, more recent approaches either upsample voxel-predictions [53] using a CRF or use different representations, like more efficient spatial subdivision”
• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, methodology): “[113] employed sparse bilateral convolution to achieve spatial-aware representation learning and multimodal (image and point cloud) reasoning.”
• PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding (2018, background): “Recent advances of 3D semantic segmentation [30, 31, 46, 19, 35, 24, 9, 39, 40, 42, 33, 7, 26, 23] have accomplished promising performance in coarse-level segmentation on the ShapeNet Part dataset [3, 45].”
• Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework (2022, result): “We compare our methods with several recent works, including SyncSpecCNN (Yi et al., 2017), SPLATNet (Su et al., 2018), etc.”
• GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “SPLATNet [16] and InterpConv [17] perform convolution on high-dimensional lattices and 3D cubes interpolated from neighboring points, respectively.”
• An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “Projection methods [3, 19, 25, 26, 59, 60, 65] project 3D data into 2D planes and convert it into 2D grids.”
• Flex-Convolution - Million-Scale Point-Cloud Learning Beyond Grid-Worlds (2018, methodology): “Further, defining splatting and slicing operations [25] has shown promising results on segmenting a facade datasets.”
• KPConv: Flexible and Deformable Convolution for Point Clouds (2019, methodology): “Several methods fall into the grid-based category, whose principle is to project the sparse 3D data on a regular structure where a convolution operation can be defined more easily [23, 28, 33].”
• Stratified Transformer for 3D Point Cloud Segmentation (2022, neutral): “…ours outperforms Method Cat. mIoU Ins. mIoU PointNet [34] 80.4 83.7 PointNet++ [35] 81.9 85.1 PCNN [2] 81.8 85.1 SpiderCNN [55] 82.4 85.3 SPLATNet [37] 83.7 85.4 DGCNN [50] 82.3 85.2 SubSparseCNN [15] 83.3 86.0 PointCNN [22] 84.6 86.1 PointConv [52] 82.8 85.7 Point2Sequence [23] - 85.2 PVCNN [27]…”
• Deep Surface Normal Estimation With Hierarchical RGB-D Fusion (2019, neutral): “, semantic segmentation [27, 5, 23], object detection [11, 20], 3D reconstruction [15, 18, 12], etc.”