LION: Latent Point Diffusion Models for 3D Shape Generation

By Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis. Neural Information Processing Systems, 2022

720 citations recorded by the source corpus.

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9793e6f3883e5ce5dbb92c0b940df06021ece1ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…models that learn directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48,…”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, positive): “…2020) are a class of generative models that have become the de-facto standard for most continuous data modalities, including images (Ramesh et al., 2021), videos (Ho et al., 2022; Singer et al., 2023), audio (Liu et al., 2023), and 3D models (Zeng et al., 2022; Poole et al., 2023; Gu et al., 2023).”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, background): “These breakthroughs have also resulted in fast progression for text-guided shape generation [9,29,53].”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “Diffusion models for image synthesis Diffusion models (DMs) [51, 53] can synthesize detailed media in many formats, such as images [34, 38], 3d shapes [66] and animations [54].”
Training Diffusion Models with Reinforcement Learning (2023, positive): “…an effective class of generative models for modalities including images (Ramesh et al., 2021; Saharia et al., 2022), videos (Ho et al., 2022; Singer et al., 2022), 3D shapes (Zhou et al., 2021; Zeng et al., 2022), and robotic trajectories (Janner et al., 2022; Ajay et al., 2022; Chi et al., 2023).”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “Zeng et al. [71] train a hierarchical auto-encoder, where the second stage encodes a point cloud of latent vectors instead of a single latent code; they employ diffusion models at both stages of the hierarchy.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “In addition to that, there are also generation methods that utilize point clouds [41, 48, 53, 74] and meshes [37] as 3D representations.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “There is a large body of work on 3D generative modeling, exploring different types of 3D representations such as 3D voxel grids [7, 12, 20, 40, 45], pointclouds [1, 21, 24, 46, 47, 49], meshes [9, 48], implicit [6, 22], or octree [15] representations.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “They aim to exploit techniques such as GAN [16], diffusion model [8, 27, 41, 78, 81] to sample a high 3D content from pre-trained distribution.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “They have driven successful applications such as text-to-image generation [57, 59, 63], natural language generation [41], time series prediction [76], audio synthesis [39], 3D shape generation [49, 87, 91], molecular conformation generation [84], protein structure generation [83], machine learning s”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, background): “…et al., 2022; Liu et al., 2023c; Zhou et al., 2021) or project the 3D models or multi-view rendered images into latent representations (Ntavelis et al., 2023; Zeng et al., 2022; Gupta et al., 2023; Jun & Nichol, 2023; Chen et al., 2023b) and perform the diffusion process in the latent space.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, positive): “Another technique, the Latent Point Diffusion (LPD) model presented in [114], generates 3D shapes with fine-grained details while controlling”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “By combing 2D image encoder and 3D generators, they generates 3D data in various representations, including 3D voxels [18, 79, 10, 81, 80, 85], point clouds [15, 88, 19, 1, 44, 90], polygon meshes [30, 73, 77, 51], and parametric models [54, 94, 95].”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “There are several DMs dealing with point cloud data [Luo and Hu 2021; Zeng et al. 2022; Zhou et al. 2021].”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Many recent works [1, 7, 10, 17, 23, 27, 30, 31, 32, 34, 42, 45, 46, 48, 75, 87, 88] try to repeat the success of diffusion models on the 3D generation task.”