Learning Gradient Fields for Shape Generation

By Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge J. Belongie, Noah Snavely, Bharath Hariharan. 2020

341 citations recorded by the source corpus.

In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized probability density, thereby moving sampled points toward the high-likelihood regions. Our model directly predicts the gradient of the log density field and can be trained with a simple objective adapted from score-based generative models. We show that our method can reach state-of-the-art performance for point cloud auto-encoding and generation, while also allowing for extraction of a high-quality implicit surface. Code is available at this https URL.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
Image Super-Resolution via Iterative Refinement (2021, background): “success in shape generation [48], and speech synthesis [33].”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “This approach, known as score matching (Hyvärinen, 2005; Vincent, 2011), has seen success in image (Song & Ermon, 2019; 2020) and shape (Cai et al., 2020) generation.”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, background): “…of generative models called Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al., 2015) has started to prove its capability to model complex data distributions such as images (Ho et al., 2020), shapes (Cai et al., 2020), graphs (Niu et al., 2020), handwriting (Luh-man & Luhman, 2020).”
Palette: Image-to-Image Diffusion Models (2021, background): “2015; Song and Ermon 2020] have received a surge of recent interest [Austin et al. 2021; Cai et al. 2020; Hoogeboom et al. 2021; Kingma et al. 2021; Song et al. 2021; Vahdat et al. 2021], resulting in several key advances in modeling continuous data.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…including unconditional image generation [Song and Ermon, 2019, Ho et al., 2020, Song et al., 2021, Nichol and Dhariwal, 2021], shape generation [Cai et al., 2020], text-to-speech [Chen et al., 2021, Kong et al., 2020] and single image super-resolution [Saharia et al., 2021, Li et al., 2021].”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “This line of work has been successfully applied to modeling high dimensional natural images (Dhariwal & Nichol, 2021; Saharia et al., 2021), audio (Kong et al., 2020), 3D point cloud (Cai et al., 2020; Zhou et al., 2021), and discrete data (Hoogeboom et al., 2021).”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, methodology): “Early 3D generation works rely on either GAN [17] or VAE [29] to model the distribution of 3D shape representation like voxel grids [6, 70], point clouds [1, 7, 31, 72], mesh [33, 63] and implicit neural representation [42, 60].”
Crystal Diffusion Variational Autoencoder for Periodic Material Generation (2021, background): “Diffusion models are a new class of generative models that have recently shown great success in generating high-quality images (Dhariwal & Nichol, 2021), point clouds (Cai et al., 2020; Luo & Hu, 2021), and molecular conformations (Shi et al., 2021).”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, background): “In parallel, Denoising Diffusion Probabilistic Models (DDPMs) [2, 7, 35, 67, 72, 77] have demonstrated impressive generative capabilities, level of details, and diversity on top of GAN [26, 61, 62, 71, 73, 76].”
HyperCube: Implicit Field Representations of Voxelized 3D Models (2021, neutral): “In such a case the implicit occupancy [1, 26, 27], distance field [28, 29] and surface parametrization [30, 31, 32, 33] models use a neural network to represent a 3D object.”
Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations (2022, background): “, 2021), and point clouds (Cai et al., 2020; Luo & Hu, 2021), the graph generation task remains to be underexplored due to the discreteness of the data structure and the complex dependency between nodes and edges.”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, result): “Flow-based [64] and gradient field-based models [4] also yield impressive results.”
DreamFusion: Text-to-3D using 2D Diffusion (2022, background): “, 2018) and point clouds (Yang et al., 2019; Cai et al., 2020; Zhou et al., 2021), but the 3D data needed is relatively scarce compared to plentiful 2D images.”
On Maximum Likelihood Training of Score-Based Generative Models (2021, background): “…et al., 2020; Song et al., 2021] is an emerging approach that has recently achieved state-of-the-art sample quality in a number of tasks, including image generation [Ho et al., 2020; Song et al., 2021], audio synthesis [Chen et al., 2020; Kong et al., 2020], and shape generation [Cai et al., 2020].”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “ShapeGF [45] learns distributions over gradient fields that model shape surfaces.”
How to Train Your Energy-Based Models (2021, background): “1), audio synthesis (Chen et al., 2020; Kong et al., 2020), and shape generation (Cai et al., 2020).”