RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

By Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen and 1 more. Computer Vision and Pattern Recognition, 2022

393 citations recorded by the source corpus.

This paper presents a 3D diffusion model that automatically generates 3D digital avatars represented as neural radiance fields (NeRFs). A significant challenge for 3D diffusion is that the memory and processing costs are prohibitive for producing high-quality results with rich details. To tackle this problem, we propose the roll-out diffusion network (RODIN), which takes a 3D NeRF model represented as multiple 2D feature maps and rolls out them onto a single 2D feature plane within which we perform 3D-aware diffusion. The RODIN model brings much-needed computational efficiency while preserving the integrity of 3D diffusion by using 3D-aware convolution that attends to projected features in the 2D plane according to their original relationships in 3D. We also use latent conditioning to orchestrate the feature generation with global coherence, leading to high-fidelity avatars and enabling semantic editing based on text prompts. Finally, we use hierarchical synthesis to further enhance details. The 3D avatars generated by our model compare favorably with those produced by existing techniques. We can generate highly detailed avatars with realistic hairstyles and facial hair. We also demonstrate 3D avatar generation from image or text, as well as text-guided editability.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7e993a9ca01dcd4538362454aaac29a18a63c000

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “More Implementation Details”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, background): “There exists another group of works that endeavor to directly produce 3D geometries like point clouds [41, 45, 75, 80], meshes [16, 37], neural fields [1, 4, 8, 15, 17, 21, 26– 28, 44, 46, 65, 76] via network inference to avoid time-consuming per-shape optimization.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, background): “Some works attempt to train text-conditioned diffusion models on 3D representations such as point clouds and volumes [1,5,9,16,26,33,35,53,58,61].”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Concurrent with Dreamfusion, SJC (Wang et al., 2023a) proposed a similar technique using the publicly available stable-diffusion model (Rombach et al., 2022).”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “To mitigate this, we narrowed the parameter sampling range from MVDream’s [15 , 60] , [0 , 30] for camera FoV and elevation to [45 , 50] and [0 , 5] , respectively, a range more typical for a generated user photos.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, background): “Wang et al. [66] pursue a related approach, jointly training separate NeRFs for every sample in a dataset, but share a subset of the parameters to ensure that all resulting models use an aligned representation space.”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “With the rise of diffusion models, pioneer works have investigated image-conditioned 3D generative modeling on various representations, e.g ., point clouds [28, 32, 47, 55, 62], meshes [1, 25], SDF grids [6, 7, 44, 60] and neural fields [11, 16, 27, 31, 59].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, background): “Automatic 3D content creation [43, 18, 33, 44] powered by large language models has drawn significant attention recently, due to its convenience to entertaining and gaming industries, virtual/augmented reality, and robotic applications.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “However, directly training a generalizable 3D diffusion model [30, 45, 46, 75] usually requires a large amount of 3D data while existing 3D datasets are insufficient for capture the complexity of arbitrary 3D shapes.”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “…diffusion models have demonstrated superior performance across a range of generation tasks in various modalities, including text-to-image generation [40, 43, 41, 17], image manipulation [26, 34, 4, 56], video synthesis [18, 47, 22], text generation [28, 16, 59], 3D avatar synthesis [39, 53], etc.”
Promises and challenges of generative artificial intelligence for human learning (2024, neutral): “The knowledge gained is usually new to the learner, fostering deep understanding through exploration.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Another line of research [26, 52, 59, 56] projects the input image into the latent space of the pretrained 3D-aware generative net-”