4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

By Christopher Bongsoo Choy, JunYoung Gwak, Silvio Savarese. Computer Vision and Pattern Recognition, 2019

2,418 citations recorded by the source corpus.

In many robotics and VR/AR applications, 3D-videos are readily-available input sources (a sequence of depth images, or LIDAR scans). However, in many cases, the 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose generalized sparse convolutions that encompass all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and trilateral-stationary conditional random fields that enforce spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that a convolutional neural network with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise and outperform the 3D convolutional neural network.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1954a5ef37030002574f1b000cc1192f3bd4cad1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds (2021, background): “Recently, to address the above issues, sparse representations [43, 10, 7] are employed to obtain smaller grids with better performance.”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, background): “Recently representation learning on point clouds has drawn lots of attention for improving the performance of point cloud classification and segmentation [10], [18], [19], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Despite its accuracy, the vast number of voxels poses a great challenge to computation and requires specialized techniques like sparse convolution [13].”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “Alternatively, point clouds can be voxelized into 3D grids and then powerful 3D CNNs are applied in [14, 28, 10, 39, 9].”
Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “Abundant methods [7, 15, 16, 34, 35, 41, 52, 61, 62] have explored 3D point cloud segmentation and obtained decent performance.”
Point Transformer V2: Grouped Vector Attention and Partition-based Pooling (2022, methodology): “Those voxel-based methods suffer from inefﬁciency because of the sparsity of point clouds until the introduction and implementation of sparse convolution [17, 18].”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, methodology): “Recently, the introduction of sparse 3D convolutions [15,9] enabled significant performance improvements by leveraging a structured space representation in a sparse fashion to operate efficiently at high resolutions.”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, methodology): “3D Semantic Segmentation We compare 4 methods for 3D semantic segmentation on ScanNet++: point-based methods PointNet [30] and PointNet++ [31] and KP-Conv [42], and sparse-voxel-based MinkowskiNet [8] on the top 78 semantic classes.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “While volumetric approaches [26, 25, 21] employ Spatially-Sparse Convolutional Networks (SSCN) [12, 2] to omit computations on inactive voxels using the sparse convolution technique.”
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts (2020, methodology): “In the past year, sparse convolutional networks [20, 12] stand out as a promising approach to standardize deep learning for point clouds, due to its computational efﬁciency and state-of-the-art performance for 3D scene understanding tasks [12, 25, 34].”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “Our model, called Mask3D, builds on recent advances in both Transformers [5, 37] and 3D deep learning [8, 17, 57]: to compute strong point features, we leverage a sparse convolutional feature backbone [8] that efficiently”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “The architecture consists of a sparse convolutional backbone based on the MinkowskiUNet [8], and a transformer decoder.”
A System for General In-Hand Object Re-Orientation (2021, methodology): “Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3D version of IMPALA-Net with sparse convolutions (Sparse3D-IMPALA-Net)1.”
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, background): “Some other approaches [30, 43, 47, 39, 13, 7] align and voxelize point cloud to produce regular 3D ordered tensors for 3D convolution.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, background): “Kernel shape adaption methods [8, 10, 13, 64] adapt the effective receptive fields of networks.”
ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes (2020, background): “In the context of 3D scene understanding, there have also been promising results on semantic and instance segmentation [53, 4, 9].”
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020, methodology): “state-of-the-art reported in [9].”
SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration (2020, background): “Their scores are even lower than the traditional methods such as FPFH, primarily because both D3Feat and FCGF have large numbers of parameters and tend to overfit the KITTI dataset, without learning the representative and general local patterns that can be applicable to the unseen dataset.”