Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

By Rui Chen, Yong Chen, Ningxin Jiao, K. Jia. IEEE International Conference on Computer Vision, 2023

814 citations recorded by the source corpus.

Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/0cbb518c364067200476a51e5ce7476a4f582770

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Control and Systems Engineering

Citations in the corpus

MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Following works have been further improving the 3D representations (Lin et al., 2023a; Tsalicoglou et al., 2023; Tang et al., 2023a; Chen et al., 2023), sampling schedules (Huang et al., 2023), and loss design (Wang et al., 2023c).”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, background): “Fantasia3D [6] explores to directly train DMTet with disentangled geometry and appearance generation.”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “Fantasia3D [4] optimizes a mesh with DMTet [45] from scratch.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, background): “…proposes Score Distillation Sampling (SDS) to address the 3D data limitation by distilling 3D geometry and appearance from powerful 2D diffusion models (Saharia et al., 2022), which inspires the development of recent 2D lifting methods (Lin et al., 2023; Wang et al., 2023b; Chen et al., 2023c).”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “…of 2D generative models (e.g. DALL-E [54] and Stable Diffusion [56]) with impressive and high-quality image generation, a large number of studies [7, 33, 40, 50] incorporate 2D diffusion and CLIP priors into per-shape optimization via various score distillation sampling strategy [50, 62, 66], or…”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Research in this space has aimed to improve distillation strategies [18, 19, 20, 21, 22], swap in other 3D representations [23, 24, 25, 26, 27], and amortize the optimization process [28].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “The developed methods improved the underlying 3D representation [43, 11, 84] and 3D consistency of the supervision [46, 77]; explored task-specific priors [28, 33, 70] and additional controls [54].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “More recently, some works (Poole et al., 2022; Wang et al., 2023b; Lin et al., 2023; Chen et al., 2023c) optimize 3D representations using score distillition loss (SDS) based on a pretrained 2D diffusion models.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Subsequent studies have focused on enhancing 3D representations [6, 17, 42, 43], refining sampling schedules [13], and optimizing loss designs [46].”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “…al. 2023b; Zhu et al. 2024], ranging from DMTet [Shen et al. 2021] to the most recent 3D Gaussian splatting [Kerbl et al. 2023], Various modifications managed to elevate the performance [Chen et al. 2023a; Li et al. 2024; Metzer et al. 2023; Seo et al. 2024; Wang et al. 2023; Zhang et al. 2023a].”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “A growing body of research [5, 31, 47, 63, 67], such as DreamField [24], DreamFusion [47], and Magic3D [31], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sampling (SDS) [47].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, methodology): “This stimulates the rapid developments of 3D content generation approaches [5, 14, 16, 21–24, 28, 30, 33, 34, 40, 45].”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “Later methods [6, 31, 32, 69, 77] build on DreamFusion and further improve the quality of 3D generation.”
Text-to-3D using Gaussian Splatting (2023, background): “In contrast, previous state-of-the-art methods (Tang, 2022; Poole et al., 2023; Lin et al., 2023; Guo et al., 2023; Chen et al., 2023) struggle in generating collapsed geometry under the same guidance and prompt, which underscores the effectiveness of our approach.”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “Besides, using mesh representation can also bring convenience to applying additional post-processing steps to enhance the results, such as SDS optimization [3, 20] or texture baking [22].”
Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “2023; Zhang et al, 2023), and 3D object generation (Wang et al, 2023a; Chen et al, 2023b; Zhou et al, 2021; Shue et al, 2023), etc. Compared to these successful applications in images, generating high-quality videos still faces significant challenges, which can be primarily attributed to the…”