3D Shape Generation and Completion through Point-Voxel Diffusion

By Linqi Zhou, Yilun Du, Jiajun Wu. 2021

707 citations recorded by the source corpus.

We propose a novel approach for probabilistic generative modeling of 3D shapes. Unlike most existing models that learn to deterministically translate a latent vector to a shape, our model, Point-Voxel Diffusion (PVD), is a unified, probabilistic formulation for unconditional shape generation and conditional, multi-modal shape completion. PVD marries denoising diffusion models with the hybrid, point-voxel representation of 3D shapes. It can be viewed as a series of denoising steps, reversing the diffusion process from observed point cloud data to Gaussian noise, and is trained by optimizing a variational lower bound to the (conditional) likelihood function. Experiments demonstrate that PVD is capable of synthesizing high-fidelity shapes, completing partial point clouds, and generating multiple completion results from single-view depth scans of real objects.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Generative Models

In 3D Vision & Pose

Citations in the corpus

On Fast Sampling of Diffusion Probabilistic Models (2021, background): “D point cloud generation (Luo and Hu, 2021; Zhou et al., 2021).”
Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For objects, methods directly output full point clouds [21,79,80] or deploy generative models [76, 84], but are typically tied to a fixed resolution.”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “This line of work has been successfully applied to modeling high dimensional natural images (Dhariwal & Nichol, 2021; Saharia et al., 2021), audio (Kong et al., 2020), 3D point cloud (Cai et al., 2020; Zhou et al., 2021), and discrete data (Hoogeboom et al., 2021).”
Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022, positive): “Point-Voxel Diffusion (PVD) (Zhou et al., 2021) is a 3D diffusion model for generating shapes from the ShapeNet dataset.”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, background): “[40] and attracts much attention recently due to state-of-the-art performance in various generation tasks including image generation [5,8,14,31], 3D point cloud generation [26,55], and audio generation [4,18,34].”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “With the rise of diffusion models, pioneer works have investigated image-conditioned 3D generative modeling on various representations, e.g ., point clouds [28, 32, 47, 55, 62], meshes [1, 25], SDF grids [6, 7, 44, 60] and neural fields [11, 16, 27, 31, 59].”
A Survey on Generative Diffusion Models (2022, background): “For instance, works such as [147], [150], [151] directly generate point clouds for 3D objects.”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “Prior work leverages 3D datasets to synthesize shapes in the form of point clouds, voxel grids, triangle meshes, and implicit functions using generative models like GANs [4, 21, 57, 65].”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “To generate point clouds with diffusion, we extend the framework used by Zhou et al. (2021a) to include RGB colors for each point in a point cloud.”
Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “2023; Zhang et al, 2023), and 3D object generation (Wang et al, 2023a; Chen et al, 2023b; Zhou et al, 2021; Shue et al, 2023), etc. Compared to these successful applications in images, generating high-quality videos still faces significant challenges, which can be primarily attributed to the…”
Diffusion Models in Vision: A Survey (2022, methodology): “[142] shape generation and completion unconditional, conditional shape completion DDPM ShapeNet, PartNet”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, background): “For example, Shap-E (Jun & Nichol, 2023) encodes each 3D shape into a set of parameters of an implicit function, and then trains an conditional diffusion models on the parameters.”
Training Diffusion Models with Reinforcement Learning (2023, positive): “We evaluate four reward functions: compressibility and in-compressibility, as determined by the JPEG compression algorithm; aesthetic quality, as determined by the LAION aesthetic quality predictor (Schuhmann, 2022); and prompt-image alignment, as determined by the LLaVA VLM (Liu et al., 2023).”
Compositional Visual Generation with Composable Diffusion Models (2022, background): “While diffusion models have been developed for image generation [39], they have further proven successful in the generation of waveforms [3], 3D shapes [47], decision making [16] and text [1], suggesting that our proposed composition operators may further be applied in such domains.”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Existing methods either use pre-trained 2D text-to-image models [35, 3, 36], together with score distillation sampling [33], to generate 3D geometries [18] or synthesize novel views [33], or train a text-conditioned 3D generative model from scratch [39, 49, 17, 29].”