Learning Descriptor Networks for 3D Shape Synthesis and Analysis

By Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu. 2018

156 citations recorded by the source corpus.

This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns. The maximum likelihood training of the model follows an "analysis by synthesis" scheme and can be interpreted as a mode seeking and mode shifting process. The model can synthesize 3D shape patterns by sampling from the probability distribution via MCMC such as Langevin dynamics. The model can be used to train a 3D generator network via MCMC teaching. The conditional version of the 3D shape descriptor net can be used for 3D object recovery and 3D object super-resolution. Experiments demonstrate that the proposed model can generate realistic 3D shape patterns and can be useful for 3D shape analysis.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, neutral): “By the known connection between score matching and energy-based modeling, our work could have implications for other recent work on energy-based models [67–69, 12, 70, 13, 11, 41, 17, 8].”
Residual Energy-Based Models for Text Generation (2020, methodology): “…gradient-based MCMC approaches like Hybrid Monte Carlo (Teh et al., 2003) and Langevyn dynamics (Ranzato et al., 2007; Du & Mordatch, 2019; Xie et al., 2016; 2017; 2019; 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text applications.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, neutral): “, [70, 71, 72, 25, 22]).”
On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “[40, 39] explore an adversarial interpretation of ML learning.”
Cooperative Training of Descriptor and Generator Networks (2016, background): “It is also related to our recent papers on 3D descriptor model [40] and the spatial-temporal descriptor model [41].”
Energy-based Out-of-distribution Detection (2020, methodology): “[41, 43–45] explored using EBMs for video generation and 3D shape pattern generation.”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, methodology): “The first operates on 3D voxels [43, 14, 45, 2], a natural extension of 2D pixels.”
Residual Energy-Based Models for Text (2021, methodology): “…gradient-based MCMC approaches like Hybrid Monte Carlo (Teh et al., 2003) and Langevyn dynamics (Ranzato et al., 2007; Du and Mordatch, 2019; Xie et al., 2016, 2017, 2019, 2018; Gao et al., 2018; Nijkamp et al., 2019) are not applicable when the input is discrete like in text applications.”
On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model (2019, background): “Recently in [50, 51], the maximum likelihood learning algorithm has been understood as an adversarial scheme or herding [46].”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[39] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Zhu Song-Chun, and Ying Nian Wu.”
Generalized Energy Based Models (2020, methodology): “In (Xie et al., 2018b;a), two independent models, a full support EBM and a generator network, are trained cooperatively using MCMC.”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, background): “For instance, 3D generative models have been explored for point clouds [2, 21], voxel grids [20, 42, 47], meshes [51], signed distance functions (SDFs) [9, 11, 25], etc.”
Improved Contrastive Divergence Training of Energy Based Models (2020, methodology): “…issue, a separate line of work utilizes an auxiliary network to amortize the negative portions of the sampling procedure (Kim & Bengio, 2016; Kumar et al., 2019; Han et al., 2019; Xie et al., 2018a; Song & Ou, 2018; Dai et al., 2019; Che et al., 2020; Grathwohl et al., 2020a; Arbel et al., 2020).”
SE(3)-Equivariant Energy-based Models for End-to-End Protein Folding (2021, methodology): “Energy-based models have been applied in various domains, including image generation [43, 9, 34, 35, 36], 3D shape pattern generation [44, 45], and molecular conformation prediction [16, 8].”
Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling (2020, background): “…without reliance on a sampler. made EBMs an appealing candidate for applications in physics (No ´ e et al., 2019), biology (Ingraham et al., 2019), neuroscience (Scellier & Bengio, 2017), and computer vision (LeCun et al., 2007; Osadchy et al., 2007; Xie et al., 2016; 2019; 2018), to name a few.”