Learning a Hierarchical Latent-Variable Model of 3D Shapes

By Shikun Liu, C. Lee Giles, Alexander Ororbia. 2017

66 citations recorded by the source corpus.

We propose the Variational Shape Learner (VSL), a generative model that learns the underlying structure of voxelized 3D shapes in an unsupervised fashion. Through the use of skip-connections, our model can successfully learn and infer a latent, hierarchical representation of objects. Furthermore, realistic 3D objects can be easily generated by sampling the VSL's latent probabilistic manifold. We show that our generative model can be trained end-to-end from 2D images to perform single image 3D model retrieval. Experiments show, both quantitatively and qualitatively, the improved generalization of our proposed model over a range of tasks, performing better or comparable to various state-of-the-art alternatives.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Gerard Pons-Moll
2. Julian Chibane
3. Mohammed Bennamoun
4. Thiemo Alldieck
5. Hamid Laga
6. Xian-Feng Han
7. Jiwen Lu
8. Jacob Walker

In Computer Graphics and Computer-Aided Design

1. Julian Chibane
2. Thiemo Alldieck
3. Sander Dieleman
4. Aäron van den Oord
5. Ali Razavi
6. Gerard Pons-Moll
7. Jiwen Lu
8. Mohammed Bennamoun

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Andrew Zisserman
5. Jitendra Malik
6. Trevor Darrell
7. Xiaogang Wang
8. Abhinav Kumar Gupta

In Generative Models

1. Sander Dieleman
2. Karen Simonyan
3. Ali Razavi
4. Aäron van den Oord
5. Jacob Walker
6. Jonathan Ho
7. Jiwen Lu
8. Jie Zhou

In 3D Vision & Pose

1. Gerard Pons-Moll
2. Julian Chibane
3. Jiwen Lu
4. Mohammed Bennamoun
5. Thiemo Alldieck
6. Faez Ahmed
7. Amin Heyrani Nobari
8. Lyle Regenwetter

Citations in the corpus

• The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Beyond raw audio generation, several generative models that capture large-scale structure have been proposed for images [2, 3, 12, 27], dialogue [40], 3D shapes [30] and symbolic music [37].”
• Predicting Video with VQVAE (2021, background): “Apart from the domain of images, deep generative models have also shown promise in other data domains such as music [10, 9], speech synthesis [37], 3D voxels [26, 32], and text [40].”
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds (2020, background): “Net40 and ModelNet10. yindicates that the model is trained on ShapeNet. Method Input Accuracy MN40 MN10 TL Network [15] voxel 74.40 - VConv-DAE [40] voxel 75.50 80.50 3DGAN [51] voxel 83.30 91.00 VSL [29] voxel 84.50 91.00 VIPGAN [17] views 91.98 94.05 LGANy[1] points 85.70 95.30 LGAN [1] points 87.”
• Deep Generative Models in Engineering Design: A Review (2021, methodology): “[95] propose the use of a Genetic Algorithm (GA) to optimize latent space design embeddings of a trained VAE variant called the Variational Shape Learner (VSL) [145], which they demonstrate on the optimization of 3D aircraft models.”
• Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Like us, the authors of [40] observe that the 1D vector is too restrictive to generate shapes with global and local structures.”
• Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “In particular, recent works tried to directly regress the 3D voxel grid [8], [11], [13], [18] without intermediation.”