3D Neural Field Generation Using Triplane Diffusion

By J. Shue, Eric Chan, Ryan Po, Zachary Ankner, Jiajun Wu, Gordon Wetzstein. Computer Vision and Pattern Recognition, 2022

336 citations recorded by the source corpus.

Diffusion models have emerged as the state-of-the-art for image generation, among other tasks. Here, we present an efficient diffusion-based model for 3D-aware generation of neural fields. Our approach pre-processes training data, such as ShapeNet meshes, by converting them to continuous occupancy fields and factoring them into a set of axis-aligned triplane feature representations. Thus, our 3D training scenes are all represented by 2D feature planes, and we can directly train existing 2D diffusion models on these representations to generate 3D neural fields with high quality and diversity, outperforming alternative approaches to 3D-aware generation. Our approach requires essential modifications to existing triplane factorization pipelines to make the resulting features easy to learn for the diffusion model. We demonstrate state-of-the-art results on 3D generation on several object classes from ShapeNet.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design

View Paper: https://www.semanticscholar.org/paper/69d8fbd6721a490ca58116242274d642e3a9bbd9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Matthias Nießner
2. Angela Dai
3. Bo Dai
4. Alexey Artemov
5. Linning Xu
6. Rui Chen
7. Peter Wonka
8. Dahua Lin

In Computer Graphics and Computer-Aided Design

1. Angela Dai
2. Matthias Nießner
3. Peter Wonka
4. Bo Dai
5. Linning Xu
6. Rui Chen
7. Dahua Lin
8. Jiapeng Tang

Citations in the corpus

• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Several generative works [8, 40] also capitalized on triplane structure to model a 3D latent space for better geometry consistency.”
• MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Depending on the chosen 3D representation, various diffusion models have been introduced, including those using Tri-plane (Shue et al., 2023; Wang et al., 2023b) or Feature grid (Karnewar et al.”
• Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Existing methods either use pre-trained 2D text-to-image models [35, 3, 36], together with score distillation sampling [33], to generate 3D geometries [18] or synthesize novel views [33], or train a text-conditioned 3D generative model from scratch [39, 49, 17, 29].”
• Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “2023; Zhang et al, 2023), and 3D object generation (Wang et al, 2023a; Chen et al, 2023b; Zhou et al, 2021; Shue et al, 2023), etc. Compared to these successful applications in images, generating high-quality videos still faces significant challenges, which can be primarily attributed to the…”
• ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Recent 3D diffusion models, specifically for tri-planes [38, 45] and feature grids [15], have emerged.”
• MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, methodology): “Diffusion-based neural field synthesis in MLP weight spaces [13] and tri-planes [55] have also been explored, alongside leveraging image-based models for optimizing NeRFs [25, 34, 48, 63].”
• 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, methodology): “The TriplaneDiffusion [Shue et al. 2022] trains an autodecoder first for each shape.”
• A Survey on Generative Diffusion Models (2022, neutral): “[24]/PVD[173]/PDR[25]/LION[201] Latent-NeRF[202]/3D-LDM[203]/[204]”