Multi-view Convolutional Neural Networks for 3D Shape Recognition
By Hang Su, Subhransu Maji, Evangelos Kalogerakis, Eric Learned-Miller. 2015
3,623 citations recorded by the source corpus.
A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.
Subfields: Computer Vision and Pattern Recognition, Geology, Industrial and Manufacturing Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Trevor Darrell
- Andrea Vedaldi
- Andrew Zisserman
In Geology
- Leonidas J. Guibas
- Matthias Nießner
- Charles Ruizhongtai Qi
- Or Litany
- Armand Joulin
- Rohit Girdhar
- Ishan Misra
- Angela Dai
In Industrial and Manufacturing Engineering
- Leonidas J. Guibas
- Hao Su
- Charles Ruizhongtai Qi
- Michael M. Bronstein
- Yue Gao
- Yifan Feng
- Matthias Nießner
- Angela Dai
Citations in the corpus
- Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, background): “, ShapeNet [29, 4], some interesting attempts [5, 24, 17] have been made on object recognition and synthesis based on voxelized 3D shape data.”
- Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs (2016, background): “…shape analysis applications, the key advantage of our approach is that it is intrinsic and thus deformation-invariant by construction , as opposed to Euclidean models [51, 59, 56, 37] that in general require significantly higher complexity and huge training sets to learn the deformation invariance.”
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2016, methodology): “Spectral CNNs: Some latest works [5, 17] use spectral CNNs on meshes.”
- Set Transformer (2018, methodology): “Pooling architectures for permutation invariant mappings Pooling architectures for sets have been used in various problems such as 3D shape recognition (Shi et al., 2015; Su et al., 2015), discovering causality (Lopez-Paz et al.”
- Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “, using multiple view images [39, 48, 46, 22], or applying convolutions on 3D voxel grids [27, 43].”
- Geometric Deep Learning: Going beyond Euclidean data (2016, background): “as range images [98], [99] or rasterized volumes [100], [101].”
- OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While performance gains can be obtained using orthogonal approaches such as network ensembles [5] or a combination of 3D and 2D convolutional networks [19,41], in this paper we deliberately focus on “pure” 3D convolutional network approaches to isolate the effect of resolution from other influencing”
- Volumetric and Multi-view CNNs for Object Classification on 3D Data (2016, methodology): “Although the multi-view CNN presented by [29] produces compelling results, we are able to improve its performance through a multi-resolution extension with improved data augmentation.”
- Dynamic Graph CNN for Learning on Point Clouds (2018, methodology): “As a simple way to overcome this issue, view-based [Su et al. 2015; Wei et al. 2016] and volumetric representations [Klokov and Lempitsky 2017; Maturana and Scherer 2015; Tatarchenko et al.”
- Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (2015, methodology): “For 3D CAD model classification, [26] and [20] took a view-based deep learning approach by rendering 3D shapes as 2D image(s).”
- Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
- Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 42, 32, 12, 39, 34], such as decomposing image silhouettes [23? ], Light Field Descriptors [2], and 2D panoramic mapping [38].”
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “…can be used for 3D shape classification [Su et al., 2015a, Sharma et al., 2016, Maturana and Scherer, 2015], 3D shape retrieval [Shi et al., 2015, Su et al., 2015a], and single image 3D reconstruction [Kar et al., 2015, Bansal et al., 2016, Girdhar et al., 2016], mostly with full supervision.”
- Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2016, methodology): “71% accuracy, which is also better than any previously published result; we report our ModelNet10 accuracy for models trained on ModelNet40 for consistency with previous results[2][3][4].”