Volumetric and Multi-view CNNs for Object Classification on 3D Data

By Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, Leonidas J. Guibas. 2016

1,648 citations recorded by the source corpus.

3D shape models are becoming widely available and easier to capture, making available 3D information crucial for progress in object classification. Current state-of-theart methods rely on CNNs to address this problem. Recently, we witness two types of CNNs being developed: CNNs based upon volumetric representations versus CNNs based upon multi-view representations. Empirical results from these two types of CNNs exhibit a large gap, indicating that existing volumetric CNN architectures and approaches are unable to fully exploit the power of 3D representations. In this paper, we aim to improve both volumetric CNNs and multi-view CNNs according to extensive analysis of existing approaches. To this end, we introduce two distinct network architectures of volumetric CNNs. In addition, we examine multi-view CNNs, where we introduce multiresolution filtering in 3D. Overall, we are able to outperform current state-of-the-art methods for both volumetric CNNs and multi-view CNNs. We provide extensive experiments designed to evaluate underlying design choices, thus providing a better understanding of the space of methods available for object classification on 3D data.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, methodology): “, ShapeNet [29, 4], some interesting attempts [5, 24, 17] have been made on object recognition and synthesis based on voxelized 3D shape data.”
Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs (2016, background): “…shape analysis applications, the key advantage of our approach is that it is intrinsic and thus deformation-invariant by construction , as opposed to Euclidean models [51, 59, 56, 37] that in general require signiﬁcantly higher complexity and huge training sets to learn the deformation invariance.”
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2016, methodology): “Volumetric CNNs: [25, 15, 16] are the pioneers applying 3D convolutional neural networks on voxelized shapes.”
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017, methodology): “We also follow [21] to randomly rotate objects for ModelNet40 data augmentation.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45, 55, 63] and generative [9, 23, 58, 64, 75, 77] 3D tasks.”
RealPoint3D: Point Cloud Generation from a Single Image with Complex Background (2018, methodology): “(Maturana and Scherer 2015) and (Qi et al. 2016) are two pioneers of using voxel-based methods for object detection and classiﬁcation.”
Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “However, this dataset is designed for volume-based object classiﬁcation [31], and there are quite few techniques that report their results with this data.”
Geometric Deep Learning: Going beyond Euclidean data (2016, background): “Many machine learning techniques successfully working on images were tried “as is” on 3D geometric data, represented for this purpose in some way “digestible” by standard frameworks, e.g. as range images [90], [91] or rasterized volumes [92], [93].”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, methodology): “When training data is synthetic and test is performed on real data, there is also a significant discrepancy of test performance, as data characteristics, such as noise and occlusions patterns, are inherently different.”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “Most existing 3D network architectures [8,29,34,47] replace the 2D pixel array by its 3D analogue, i.”
Dynamic Graph CNN for Learning on Point Clouds (2018, neutral): “2015]—or their combination [Qi et al. 2016]—“place” geometric data onto a grid.”
PCPNet Learning Local Shape Properties from Raw Point Clouds (2017, background): “Many learning-based approaches are aimed at estimating global properties for tasks such as shape classiﬁcation and often are based either on 2D projections (views) of 3D shapes or on volumetric representations (see, e.g., [QSN ∗ 16] for a comparison).”
Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, methodology): “Table 2 compares our result to several recent works, based either on volumetric [40, 26, 1, 30] or rendered image representation [38].”
Multi-view 3D Object Detection Network for Autonomous Driving (2016, background): “[20, 15, 10] investigate volumetric and multi-view representation of point cloud for 3D object classification.”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 42, 32, 12, 39, 34], such as decomposing image silhouettes [23? ], Light Field Descriptors [2], and 2D panoramic mapping [38].”
SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis (2017, methodology): “Deep learning has also been successfully applied to other 3D applications like volumetric shape retrieval or object classification [28, 25, 16, 19].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “By utilizing large datasets of 3D models and high capacity 3D convolutional neural networks (3D CNNs), our approach learns to smooth out sensor noise, deals with outliers and completes missing 3D geometry.”
3D shape segmentation via shape fully convolutional networks (2017, methodology): “By using volumetric representation [27, 28], standard CNN 2d convolution and pooling methods can be easily extended to 3D convolution and pooling.”
Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, methodology): “In the ﬁrst variant, we train the 3D-CNN of Qi et al. [31] on partial input to reﬂect the occlusion patterns of the test data.”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “We use ModelNet [Wu et al., 2015] for testing, following Sharma et al. [2016], Maturana and Scherer [2015], Qi et al. [2016].”