Point-Voxel CNN for Efficient 3D Deep Learning

By Zhijian Liu, Haotian Tang, Yujun Lin, Song Han. Neural Information Processing Systems, 2019

855 citations recorded by the source corpus.

We present Point-Voxel CNN (PVCNN) for efficient, fast 3D deep learning. Previous work processes 3D data using either voxel-based or point-based NN models. However, both approaches are computationally inefficient. The computation cost and memory footprints of the voxel-based models grow cubically with the input resolution, making it memory-prohibitive to scale up the resolution. As for point-based networks, up to 80% of the time is wasted on structuring the irregular data which have rather poor memory locality, not on the actual feature extraction. In this paper, we propose PVCNN that represents the 3D input data in points to reduce the memory consumption, while performing the convolutions in voxels to largely reduce the irregular data access and improve the locality. Our PVCNN model is both memory and computation efficient. Evaluated on semantic and part segmentation datasets, it achieves much higher accuracy than the voxel-based baseline with 10x GPU memory reduction; it also outperforms the state-of-the-art point-based models with 7x measured speedup on average. Remarkably, narrower version of PVCNN achieves 2x speedup over PointNet (an extremely efficient model) on part and scene segmentation benchmarks with much higher accuracy. We validate the general effectiveness of our PVCNN on 3D object detection: by replacing the primitives in Frustrum PointNet with PVConv, it outperforms Frustrum PointNet++ by 2.4% mAP on average with 1.5x measured speedup and GPU memory reduction.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fdde9a63877db44c1b6a7f5cff5d47f01465e793

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, methodology): “(Liu et al., 2019) combines both voxel CNN and point multi-layer percetron network for efﬁcient point feature learning.”
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation (2021, methodology): “One recent attempt is PVCNN [21], a point-voxel fusion scheme, in which voxel offers a coarse-grained local feature, while point preserves fine-grained geometrical features by running simple per-point MLPs.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “For point clouds (PCs) the pioneering PointNet based architectures [60, 61] spearheaded research, such as kernel point convolutions [73, 59, 33, 79], tree-based graph convolutions [68] normalizing ﬂows [85], combinations of points and voxels for eﬃciency [46] or domain adaptation techniques [62].”
Learning to Recover 3D Scene Shape from a Single Image (2020, methodology): “For parameters specific to PVCNN, such as the voxel size, we follow the original work [26].”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, neutral): “To overcome the drawbacks of both point-based meth-ods ( i.e. , irregular and sparse data access, poor memory locality [23]) and voxel-based methods ( i.e. , quantization loss), several methods [3, 16, 36, 37, 53] have started to learning from 3D point clouds using point-voxel joint representations.”
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection (2019, methodology): “[20] combines both voxelbased CNN and point-based SharedMLP for efficient point cloud feature learning.”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “Recent learning based approaches [16, 43, 37] mainly include projection-based, voxel-based and pointbased schemes which are outlined here.”
PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, background): “Early attempts on point cloud completion [6, 15, 32, 33, 24, 38, 20, 19, 53, 49, 42] try to migrate mature methods from 2D completion tasks to 3D point clouds by voxelization and 3D convolutions.”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, methodology): “Several methods [2,3,8,33] convert point clouds into 3D voxels and then apply 3D convolutions to them.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, methodology): “The encoder networks, as well as the decoder and the latent point DDM, operating on point clouds x, are all implemented based on Point-Voxel CNNs (PVCNNs) [80], following Zhou et al.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Despite these efforts, the efficiency of PointNet-based methods is still limited since they inherently need to process sparse data, which is more difficult to accelerate [27].”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, methodology): “We therefore explore a separate point-voxel representation [22, 33], and our work is most related to point-voxel CNN [24], which proposes to voxelize the point clouds for 3D convolution.”
Stratified Transformer for 3D Point Cloud Segmentation (2022, positive): “…[37] 83.7 85.4 DGCNN [50] 82.3 85.2 SubSparseCNN [15] 83.3 86.0 PointCNN [22] 84.6 86.1 PointConv [52] 82.8 85.7 Point2Sequence [23] - 85.2 PVCNN [27] - 86.2 RS-CNN [25] 84.0 86.2 KPConv [41] 85.0 86.2 InterpCNN [30] 84.0 86.3 DensePoint [24] 84.2 86.4 PAConv [54] 84 Point Transformer by a large…”
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, positive): “1 3D Generator Input Encoder We use PVCNN [34] as an input encoder to extract a 3D feature volume Fvol(x) from a point cloud.”
Fast geometric learning with symbolic matrices (2020, background): “Geometric computations present a clear gap in performances between Python and C++: notable examples are implementations of point cloud convolutions or of the nearest neighbor search [65, 76].”