Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

By Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston. 2016

626 citations recorded by the source corpus.

When working with three-dimensional data, choice of representation is key. We explore voxel-based models, and present evidence for the viability of voxellated representations in applications including shape modeling and object classification. Our key contributions are methods for training voxel-based variational autoencoders, a user interface for exploring the latent space learned by the autoencoder, and a deep convolutional neural network architecture for object classification. We address challenges unique to voxel-based representations, and empirically evaluate our models on the ModelNet benchmark, where we demonstrate a 51.5% relative improvement in the state of the art for object classification.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Existing representations can be broadly categorized into three categories: voxel-based representations [4,19,43,58,64,69,75] , point-based representations [1,17] and mesh representations [34,57,70], see Fig.”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “Our 8M parameter model achieves an accuracy of 93.28%, compared to a 93.61% accuracy from a hand-designed Inception-ResNet [5] with 18M parameters trained on the larger ModelNet40 dataset.”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While performance gains can be obtained using orthogonal approaches such as network ensembles [5] or a combination of 3D and 2D convolutional networks [19,41], in this paper we deliberately focus on “pure” 3D convolutional network approaches to isolate the effect of resolution from other influencing”
Generating 3D faces using Convolutional Mesh Autoencoders (2018, background): “Recent works have explored volumetric convolutions [8] for 3D representations.”
Dynamic Graph CNN for Learning on Point Clouds (2018, neutral): “…[Maturana and Scherer 2015] 83.0 85.9 Subvolume [Qi et al. 2016] 86.0 89.2 VRN (single view) [Brock et al. 2016] 88.98 - VRN (multiple views) [Brock et al. 2016] 91.33 - ECC [Simonovsky and Komodakis 2017] 83.2 87.4 PointNet [Qi et al. 2017b] 86.0 89.2 PointNet++ [Qi et al. 2017c] - 90.7…”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “…our ShapeHD aims to model ambiguity on the next level: even among the realistic shapes, there are still multiple shapes explaining the observation well (c). growing number of papers on exploiting shape structures and regularities [34,51], and papers on leveraging strong database priors [46,32,4].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “Generalization on Unseen Categories: Most existing approaches that leverage deep learning for 3D reconstruction train a model specific for a particular object class [4, 8, 17, 41, 35] which typically does not generalize well to other views=10 views=20”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, background): “Quite recently and parallel to this work, there have been several works utilizing 2D or 3D CNNs merely on 3D CAD models of ModelNet [38] in supervised [2, 9, 13], semi-supervised [24] and unsupervised [37] fashions.”
CubeNet: Equivariance to 3D Rotation and Translation (2018, methodology): “We also rescale the voxel values to {−1,5} instead of {0,1} as in [4], who showed it helps with sparse voxel volumnes.”
3D Object Dense Reconstruction from a Single Depth View (2018, methodology): “(1) `ae For the generator, inspired by [80], we use modified binary cross-entropy loss function instead of the standard version.”
Deep Learning Advances in Computer Vision with 3D Data (2017, result): “…the beneﬁts of 2D views and 3D data were also made leading to satisfying results (e.g., Hegde and Zadeh [2016]), however, a recently published work (Brock et al. [2016]) demonstrated the power of volumetric models achieving state-of-the-art classiﬁcation accuracy on both ModelNet40 (i.e., 95.54”
Shape Inpainting Using 3D Generative Adversarial Network and Recurrent Convolutional Networks (2017, methodology): “We also compare our algorithm with the state-of-the-art methods [26, 21, 11, 3, 23, 19, 5].”
Learning Shape Templates With Structured Implicit Functions (2019, methodology): “We argue that our structured implicit representation is superior for template learning compared to decoding voxels [6, 55, 56], sparse-voxel octrees [50], points [11], meshes [15, 22, 53], box primitives [51], signed-distance function estimators [40], or indicator function estimators [34].”
SO-Net: Self-Organizing Network for Point Cloud Analysis (2018, background): “, qi-MVCNN-MultiRes [27] and VRN Ensemble [6], are still better than SO-Net in ModelNet classification, but their multi-view / voxel grid representations are far less scalable and flexible than our point cloud representation, as illustrated in Sec.”
3D2SeqViews: Aggregating Sequential Views for 3D Global Feature Learning by CNN With Hierarchical Attention Aggregation (2019, methodology): “For fair comparison, the result of VRN [31] is presented with a single CNN, where twice more views than ours are employed, the result of RotationNet [21] is presented with views taken by the default camera system orientation which keeps identical with other methods, and the result of Spherical…”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, background): “The first operates on 3D voxels [43, 14, 45, 2], a natural extension of 2D pixels.”
SPNet: Deep 3D Object Classification and Retrieval using Stereographic Projection (2018, background): “33% VRN Ensemble [5] 678MB 93.”
Texture Fields: Learning Texture Representations in Function Space (2019, background): “Recently, the vision community has witnessed impressive progress in reconstruction tasks like single image 3D reconstruction [6, 7, 9, 13, 23, 25, 43, 45] and generating 3D objects [4, 32, 46] using learning-based techniques which resolve ambiguities by incorporating prior knowledge.”