Diffusion Probabilistic Models for 3D Point Cloud Generation

By Shitong Luo, Wei Hu. 2021

1,048 citations recorded by the source corpus.

We present a probabilistic model for point cloud generation, which is fundamental for various 3D vision tasks such as shape completion, upsampling, synthesis and data augmentation. Inspired by the diffusion process in non-equilibrium thermodynamics, we view points in point clouds as particles in a thermodynamic system in contact with a heat bath, which diffuse from the original distribution to a noise distribution. Point cloud generation thus amounts to learning the reverse diffusion process that transforms the noise distribution to the distribution of a desired shape. Specifically, we propose to model the reverse diffusion process for point clouds as a Markov chain conditioned on certain shape latent. We derive the variational bound in closed form for training and provide implementations of the model. Experimental results demonstrate that our model achieves competitive performance in point cloud generation and auto-encoding. The code is available at https://github.com/luost26/diffusion-point-cloud.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Variants of SGMs have been applied to images [1, 2, 5, 6], audio [7, 8, 9, 10], graphs [11] and point clouds [12, 13].”
On Fast Sampling of Diffusion Probabilistic Models (2021, background): “D point cloud generation (Luo and Hu, 2021; Zhou et al., 2021).”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Numerous applications have emerged notably in the image domain, such as image manipulation, translation, super-resolution [8,32,35,43], in speech and text domains [5,6], or 3D point cloud [34].”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “A function like push apart can for example be relevant in turning a multiset of random 3d points that are close together (i.e., similar) into a 3d shape with points that are farther away (Luo & Hu, 2021).”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “…Generative Models DDPM (SohlDickstein et al., 2015; Ho et al., 2020) has undergone several theoretical developments (Song et al., 2021b) and produces high quality samples in many domains (Ho et al., 2020; Dhariwal & Nichol, 2021; Chen et al., 2021a; Popov et al., 2021; Luo & Hu, 2021).”
GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (2022, background): “Recently, some other works (Luo & Hu, 2021; Chibane et al., 2020) has also been proposed for 3D structure generation with diffusion-based models, but focus on the point cloud problem.”
ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, background): “Denoising diffusion probabilistic models (DDPMs) [8, 23, 44] are likelihood-based generative models that have recently succeeded to advance the state-of-the-art results in several important domains including image synthesis [5, 44], audio synthesis [11], and 3D point cloud generation [27], and have ”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “There are several DMs dealing with point cloud data [Luo and Hu 2021; Zeng et al. 2022; Zhou et al. 2021].”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, background): “There exists another group of works that endeavor to directly produce 3D geometries like point clouds [41, 45, 75, 80], meshes [16, 37], neural fields [1, 4, 8, 15, 17, 21, 26– 28, 44, 46, 65, 76] via network inference to avoid time-consuming per-shape optimization.”
Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design (2021, methodology): “Our iterative reﬁnement model is also related to score matching methods for molecular conformation prediction (Shi et al., 2021) and diffusion-based methods for point clouds (Luo & Hu, 2021).”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “…Models Diffusion models [16, 17, 25, 26, 27, 28, 29, 30, 31] have achieved unprecedented results in various generative modeling tasks, including image/video generation [9, 32, 33, 34, 35], audio generation [36], point cloud generation [37, 38, 39, 40], biological generation [34, 41, 42, 43], etc..”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, background): “…Probabilistic Models (DDPM) (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) have emerged as a powerful family of generative models, capable of synthesizing high-quality images, audio, and 3D shapes (Ho et al., 2020; 2021; Chen et al., 2021a;b; Cai et al., 2020; Luo & Hu, 2021).”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, background): “This light-weight fine-tuning share a similar spirit as the ‘instruction fine-tuning’ (Mishra et al., 2022; Wei et al., 2021) for LLM (large language model) alignment.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Computer Vision [137] [202],[198],[147],[200],[181], Super Resolution, Inpainting, and Translation [91],[10],[174],[38],[224],[36],[161] Semantic Segmentation [9],[17],[76] Video Generation [89],[93],[268],[275] Point Cloud Completion and Generation [286],[150],[155],[143],[274] Anomaly Detection [2”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, neutral): “For 3D point cloud generation, [24] viewed points in point clouds as particles in a thermodynamic system in contact with a heat bath and then introduced a diffusion model to estimate the original point clouds distribution.”
DeepCache: Accelerating Diffusion Models for Free (2023, background): “A large number of attractive applications have been facilitated with diffusion models, including but not limited to image editing [2, 22, 40], image super-enhancing [28, 53], image-to-image translation [7, 51], text-to-image generation [43, 47, 48, 52] and text-to-3D generation [32, 37, 45].”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, background): “[40] and attracts much attention recently due to state-of-the-art performance in various generation tasks including image generation [5,8,14,31], 3D point cloud generation [26,55], and audio generation [4,18,34].”
Diffusion Probabilistic Modeling for Video Generation (2022, methodology): “Apart from the traditional computer vision tasks, diffusion models were proven to be effective in audio synthesis [68,69], while Luo and Hu [70] hybridized normalizing ﬂows and diffusion model to generative 3D point cloud samples.”