3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

By Benjamin Graham, Martin Engelcke, L. V. D. van der Maaten. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

1,854 citations recorded by the source corpus.

Convolutional networks are the de-facto standard for analyzing spatio-temporal data such as images, videos, and 3D shapes. Whilst some of this data is naturally dense (e.g., photos), many other data sources are inherently sparse. Examples include 3D point clouds that were obtained using a LiDAR scanner or RGB-D camera. Standard "dense" implementations of convolutional networks are very inefficient when applied on such sparse data. We introduce new sparse convolutional operations that are designed to process spatially-sparse data more efficiently, and use them to develop spatially-sparse convolutional networks. We demonstrate the strong performance of the resulting models, called submanifold sparse convolutional networks (SS-CNs), on two tasks involving semantic segmentation of 3D point clouds. In particular, our models outperform all prior state-of-the-art on the test set of a recent semantic segmentation competition.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1d6a5d0299ed8458191e4e0407d4d513e6a7dd7e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “Alternatively, point clouds can be voxelized into 3D grids and then powerful 3D CNNs are applied in [14, 28, 10, 39, 9].”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, methodology): “Thanks to the flexibility of our framework, we therefore easily train a parallel SCN network [11] to estimate more accurate per-point semantic labels for the predicted instances of our 3D-BoNet.”
VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (2023, background): “1× 1 submanifold sparse convolution [19] is the fully connected.”
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (2019, neutral): “6 SSC-UNet [10] 30.”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “Inspired by [59, 51], we choose the SparseConvNet [20] to efﬁciently process the structured latent codes, which outputs a latent code volume.”
PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding (2018, background): “Recent advances of 3D semantic segmentation [30, 31, 46, 19, 35, 24, 9, 39, 40, 42, 33, 7, 26, 23] have accomplished promising performance in coarse-level segmentation on the ShapeNet Part dataset [3, 45].”
Deep Hough Voting for 3D Object Detection in Point Clouds (2019, methodology): “Instead of relying on hand-crafted features, we leverage recently proposed deep networks [36, 9, 43, 25] on point clouds for point feature learning.”
CIA-SSD: Confident IoU-Aware Single-Stage Object Detector From Point Cloud (2020, methodology): “Speciﬁcally, our four blocks have { 2, 2, 3, 3 } SSC layers, respectively, and an SC layer appended to the end of each block for 2x downsampling on the 3D feature maps.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “3 SparseConvNet [21] 72.”
PointSite: a point cloud segmentation tool for identification of protein ligand binding atoms (2019, methodology): “Then, we formulate the LBS identification problem as a point clouds segmentation problem, which could be elegantly solved by Submanifold Sparse Convolutional networks (SSCN) [11].”
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (2023, methodology): “We propose two possible implementations to enable FCMAE pre-training: 1) sparse encoding using sparse convolution [15, 27, 28] supported by external libraries [15, 18], and 2) simulating sparse encoding with the masked dense convolution, which can be easily implemented by applying binary masks befor”
PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds (2021, background): “Recently, to address the above issues, sparse representations [43, 10, 7] are employed to obtain smaller grids with better performance.”
3D Object Detection with Pointformer (2020, background): “3D sparse convolution [11] is very effective on voxel grids.”
Deep Learning for 3D Point Clouds: A Survey (2019, methodology): “spherical [183], [184], [185], volumetric [166], [186], [187], permutohedral lattice [188], [189], and hybrid representations [190], [191], as shown in Fig.”
BSP-Net: Generating Compact Meshes via Binary Space Partitioning (2019, methodology): “Finally, recent methods that perform sparse convolutions [15] on voxel grids are similarly limited to encoders.”
Center-based 3D Object Detection and Tracking (2020, methodology): “VoxelNet [64] uses a PointNet [38] inside each voxel to generate a unified feature representation from which a head with 3D sparse convolutions [18] and 2D convolutions produces detections.”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, methodology): “…2D pixels (Chen et al., 2017; Yang et al., 2018b) or 3D voxels (Zhou and Tuzel, 2018; Chen et al., 2019b) for processing them with 2D/3D CNN. Sparse convolution (Graham et al., 2018) is adopted in (Yan et al., 2018; Shi et al., 2020b) to effectively learn sparse voxel features from point clouds.”
SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud (2021, methodology): “SECOND [30] modifies the sparse convolution [7, 15] to efficiently extract features from sparse voxels.”