Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors

By Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov and 1 more. International Conference on Learning Representations, 2023

475 citations recorded by the source corpus.

We present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images. Our code, models, and generated 3D assets are available at https://github.com/guochengqian/Magic123.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2605f0776dcda957fa540f88cdec53badab848b1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Text-to-3D using Gaussian Splatting (2023, background): “This development sparked a surge of interest in text-to-3D generation through image diffusion prior [15, 38, 52, 54, 74, 85].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, unknown): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “Inspired by Zero123 [25], several recent works [20, 23, 24, 26, 28, 37, 44, 48, 62] finetune image or video diffusion models to generate novel view images as a stronger guidance for 3D generation.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “…2022), to introduce semantics and multi-view guidance for image-to-3D reconstruction (Liu et al., 2023b; Tang et al., 2023; Deng et al., 2023; Shen et al., 2023b; Anciukeviˇcius et al., 2023; Melas-Kyriazi et al., 2023; Metzer et al., 2023; Xu et al., 2023; Qian et al., 2023; Li et al., 2023a).”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “Leveraging the novel view guidance provided by Zero123, recent image-to-3D methods [22, 36, 56] show improved 3D consistency and can generate plausible shapes from open-domain images.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “In Table 1, we report the CLIP-similarity (Radford et al., 2021; Qian et al., 2023; Liu et al., 2023a) and average generation time of different image-to-3D methods on a collection of images from previous works (Melas-Kyriazi et al., 2023; Liu et al., 2023a; Tang et al., 2023b) and Internet.”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, background): “More recently, some work [22, 28, 32] enable zero-shot novel view synthesis with even one input by diffusion model priors, but can hardly handle complex scenes and with lower efficiency.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “In our evaluation, we compared ImageDream’s performance against several SoTA baselines, including Zero123-XL [19] (trained on 10x larger data than ours), Magic123 [31], and SyncDreamer [20].”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Using text-based priors for single-image-to-3D has also shown promise [29, 30, 31], but requires a complex balancing of the image observation with additional constraints.”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, unknown): “Alternative solutions attempt to employ SDS to optimize a coherent neural field [Qian et al. 2024; Sun et al. 2024; Tang et al. 2024; Zhang et al. 2023d], but they generally require long optimization time.”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, background): “This quickly motivates a great number of following works [5, 16, 21, 29, 33, 35, 49] and becomes a critical integration of them.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, unknown): “Some single-image-to-3D meth-ods [13, 19, 30, 33–36, 38, 54, 60, 65, 66, 71, 78, 79, 82, 83] also employ similar ideas.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “More recently, some methods utilize data priors from pretrained 2D diffusion models to lift a single 2D image to 3D by providing supervision at novel views using SDS loss (Liu et al., 2023b; Qian et al., 2023; Melas-Kyriazi et al., 2023) or generated multi-view images (Liu et al., 2023a).”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Zero123 [40] Magic123 [50] One-2-3-45 [39] Point-E [46] Shap-E [30] Figure 6.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “For the task of text-to-3D [2, 5, 6, 23, 31, 52, 53, 61, 67, 69, 73, 82] or image-to-3D synthesis [42, 48, 50, 54, 58, 71], these methods typically optimize a 3D representation (i.e., NeRF, mesh, or SDF), and then leverage neural rendering to generate 2D images from various viewpoints.”