View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions

By Zhizhong Han, Mingyang Shang, Yu-Shen Liu, Matthias Zwicker. 2018

143 citations recorded by the source corpus.

In this paper, we present a novel unsupervised representation learning approach for 3D shapes, which is an important research challenge as it avoids the manual effort required for collecting supervised data. Our method trains an RNNbased neural network architecture to solve multiple view inter-prediction tasks for each shape. Given several nearby views of a shape, we define view inter-prediction as the task of predicting the center view between the input views, and reconstructing the input views in a low-level feature space. The key idea of our approach is to implement the shape representation as a shape-specific global memory that is shared between all local view inter-predictions for each shape. Intuitively, this memory enables the system to aggregate information that is useful to better solve the view inter-prediction tasks for each shape, and to leverage the memory as a viewindependent shape representation. Our approach obtains the best results using a combination of L2 and adversarial losses for the view inter-prediction task. We show that VIP-GAN outperforms state-of-the-art methods in unsupervised 3D feature learning on three large-scale 3D shape benchmarks.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, background): “It is also possible to treat point clouds and views as sequences [26, 17, 15], or to use unsupervised learning [16].”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, background): “Initial line of works exploit generative modelling using generative adversarial networks [1, 15, 57] and auto-encoders [11, 16, 26, 65, 71], which aims to reconstruct a given input point cloud with varying architectural designs.”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, unknown): “3D-GAN [52] 83.3 Latent-GAN [46] 85.7 SO-Net [25] 87.3 FoldingNet [57] 88.4 MAP-VAE [19] 88.4 VIP-GAN [18] 90.2 DGCNN + Jiasaw [43] 90.6 DGCNN + OcCo [49] 90.7 DGCNN + CrossPoint [2] 91.2 Transformer + OcCo [60] 89.6 Point-BERT [60] 87.4 Point-M2AE 92.9 Improvement +1.7 1 , the extra spatial noises…”
Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds (2020, methodology): “es that the model is trained on ShapeNet. Method Input Accuracy MN40 MN10 TL Network [15] voxel 74.40 - VConv-DAE [40] voxel 75.50 80.50 3DGAN [51] voxel 83.30 91.00 VSL [29] voxel 84.50 91.00 VIPGAN [17] views 91.98 94.05 LGANy[1] points 85.70 95.30 LGAN [1] points 87.27 92.18 FoldingNety[56] point”
Self-Supervised Deep Learning on Point Clouds by Reconstructing Space (2019, methodology): “In the first experiment, we follow the same procedure as in [1, 34, 37, 12].”
Unsupervised Point Cloud Pre-training via Occlusion Completion (2020, methodology): “Initial work used latent generative models such as generative adversarial networks (GANs) [1, 14, 54] and autoen-few Jigsaw [41] cTree [43] OcCo (ours) Figure 1: The relative improvement over random initialisation of multiple pre-training methods: Jigsaw [42], cTree [44], and OcCo (ours) for…”
SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer (2021, background): “In 3D computer vision [11, 18, 20, 14, 13] applications, raw point clouds captured by 3D scanners and depth cameras are usually sparse and incomplete [53, 54, 48] due to occlusion and limited sensor resolution.”