Magic3D: High-Resolution Text-to-3D Content Creation

By Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin. Computer Vision and Pattern Recognition, 2022

1,645 citations recorded by the source corpus.

DreamFusion [31] has recently demonstrated the utility of a pretrained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF) [23], achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2× faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bdf4af8311637c681904e71cf50f96fd0026f578

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “An emerging line of work has sought to apply similar methods to create 3D models from text [9, 56, 29, 41, 38], yet existing approaches come with a number of significant limitations and lack the generality of 2D text-to-image models.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, neutral): “However, optimizing NeRF can be time-consuming, despite its wide usage in 3D reconstruction [3, 6, 13, 21, 44] and generation [22, 35] tasks.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Multi-Modal Learning Text-to-Image Generation [7],[216],[231],[189],[87],[65],[229],[126],[269],[9],[299],[321] Scene Graph-to-Image Generation [306] Text-to-3D Generation [296],[155],[207] Text-to-Motion Generation [263, 322],[130] Text-to-Video Generation [245],[96],[283],[210] Text-to-Audio Gener”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, methodology): “Prior arts [32, 18] often produce models with limited diversity and excessively smooth textures.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “A growing body of research [5, 31, 47, 63, 67], such as DreamField [24], DreamFusion [47], and Magic3D [31], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sampling (SDS) [47].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “More recently, some works (Poole et al., 2022; Wang et al., 2023b; Lin et al., 2023; Chen et al., 2023c) optimize 3D representations using score distillition loss (SDS) based on a pretrained 2D diffusion models.”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, methodology): “This also helps avoid certain design decisions that result in the cartoon-ish appearance commonly seen in unconditional 3D content generation methods [33, 47, 19].”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, neutral): “Some promising directions for future research include: (cid:129) Expanding the application of diffusion models: Investigate the application of diffusion models in other domains, such as ﬁnance, healthcare, transportation, and logistics, where complex utility maximization problems often arise.”
Implicit Diffusion Models for Continuous Super-Resolution (2023, positive): “On the other hand, GAN-based and flow-based models, variational autoencoders (VAEs), and autoregressive models (ARMs) have been proposed to improve the fine details of SR images.”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, background): “DreamFusion [47] and later on Magic3D [48] demonstrated that 3D shapes could be generated from language prompts with the help of text-to-image models.”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “We also use the point lighting (Poole et al., 2023) and soft shading (Lin et al., 2023a) to regularize the geometry.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “Motivated by the great success in 2D, a series of works have significantly propelled the 3D synthesis development (Lin et al., 2023; Wang et al., 2023c) based on Score Distillation Sampling (SDS) (Poole et al.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “…variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation [1,23,33,52,57,77,91].”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Research in this space has aimed to improve distillation strategies [18, 19, 20, 21, 22], swap in other 3D representations [23, 24, 25, 26, 27], and amortize the optimization process [28].”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Inspired by these results, recent works have explored text-conditional generation in other modalities, such as video (Hong et al., 2022; Singer et al., 2022; Ho et al., 2022b;a) and 3D objects (Jain et al., 2021; Poole et al., 2022; Lin et al., 2022a; Sanghi et al., 2021; 2022).”
Text-to-3D using Gaussian Splatting (2023, methodology): “, 2023), Magic3D (Lin et al., 2023), Fantasia3D (Chen et al.”