Submanifold Sparse Convolutional Networks

By Benjamin Graham, L. V. D. van der Maaten. arXiv.org, 2017

548 citations recorded by the source corpus.

Convolutional network are the de-facto standard for analysing spatio-temporal data such as images, videos, 3D shapes, etc. Whilst some of this data is naturally dense (for instance, photos), many other data sources are inherently sparse. Examples include pen-strokes forming on a piece of paper, or (colored) 3D point clouds that were obtained using a LiDAR scanner or RGB-D camera. Standard "dense" implementations of convolutional networks are very inefficient when applied on such sparse data. We introduce a sparse convolutional operation tailored to processing sparse data that differs from prior work on sparse convolutional networks in that it operates strictly on submanifolds, rather than "dilating" the observation with every layer in the network. Our empirical analysis of the resulting submanifold sparse convolutional networks shows that they perform on par with state-of-the-art methods whilst requiring substantially less computation.

Subfields: Computational Mechanics

View Paper: https://www.semanticscholar.org/paper/11356cd6bb0f2776a88cd584ff108470414c6594

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

Citations in the corpus

Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference (2019, background): “Note that the input of a dynamic convolution is a dense matrix, making it fundamentally different from methods operating on sparse data [14, 15].”
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks (2017, background): “1These operators appeared earlier in an unpublished report [5], including experiments on several classification datasets.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, methodology): “Then, the voxel backbone in Figure 2 is utilized to process input voxels with sparse convolution [17].”
Multiresolution Tree Networks for 3D Point Cloud Processing (2018, background): “multiscale and sparse representations [33,41,42,17,38,16] at the expense of additional book keeping.”
PointCNN: Convolution On X-Transformed Points (2018, background): “The activations are kept sparse in [13] to retain sparsity in convolved layers.”
SECOND: Sparsely Embedded Convolutional Detection (2018, background): “As an alternative to normal sparse convolution, submanifold convolution [27] restricts an output location to be active if and only if the corresponding input location is active.”
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection (2019, methodology): “3D sparse convolution [6, 5] are adopted in [37, 28] to effectively learn sparse voxel-wise features from the point clouds.”
PointConv: Deep Convolutional Networks on 3D Point Clouds (2018, methodology): “SSCN [7] utilizes the convolution based on a volumetric grid with novel speed/memory improvements by considering CNN outputs only on input points.”
Kubric: A scalable dataset generator (2022, background): “We demonstrate two representative baselines for 2D image and 3D point cloud segmentation: DeepLab [15] and SparseConvNet [35], respectively.”
SBNet: Sparse Blocks Network for Fast Inference (2018, methodology): “Our approach differs from [6, 20, 25] in that we gather block-wise slices from tensors and maintain the tensor shape instead of lowering them to vectors.”
MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation (2019, background): “[3] propose an super-efﬁcient volumetric CNN based on submanifold sparse convolution [4] to process the entire point cloud of an indoor scene, which achieves promising results on the semantic segmentation task.”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, methodology): “With the help of sparse convolution [14, 61], we can significantly reduce the computation cost of detection heads on low-level features, while keeping the detection accuracy for small objects.”
Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “Abundant methods [7, 15, 16, 34, 35, 41, 52, 61, 62] have explored 3D point cloud segmentation and obtained decent performance.”
SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization (2019, background): “Researchers have also proposed to take advantage of sparsity in the activation maps [13, 25, 5, 21].”
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (2023, methodology): “We propose two possible implementations to enable FCMAE pre-training: 1) sparse encoding using sparse convolution [15, 27, 28] supported by external libraries [15, 18], and 2) simulating sparse encoding with the masked dense convolution, which can be easily implemented by applying binary masks befor”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, unknown): “To this end, Yan et al. [49] present an efficient architecture called SECOND by leveraging the 3D submanifold sparse convolution [9].”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, methodology): “3D sparse convolution [61], [81] are adopted in [16], [24] to effectively learn sparse voxel-wise features from the point”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “, 2020] proposed an object detection pipeline which utilizes a 3D sparse U-Net [Graham and van der Maaten, 2017] and a graph convolution module.”
Multi-agent Embodied Question Answering in Interactive Environments (2020, methodology): “Since the 3D volume can be huge, we use submanifold sparse convolutions [8] instead of traditional convolutions to process those sparse data.”