Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

By Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, Dong Chen. IEEE International Conference on Computer Vision, 2023

420 citations recorded by the source corpus.

In this work, we investigate the problem of creating high-fidelity 3D content from only a single image. This is inherently challenging: it essentially involves estimating the underlying 3D geometry while simultaneously hallucinating unseen textures. To address this challenge, we leverage prior knowledge from a well-trained 2D diffusion model to act as 3D-aware supervision for 3D creation. Our approach, Make-It-3D, employs a two-stage optimization pipeline: the first stage optimizes a neural radiance field by incorporating constraints from the reference image at the frontal view and diffusion prior at novel views; the second stage transforms the coarse model into textured point clouds and further elevates the realism with diffusion prior while leveraging the high-quality textures from the reference image. Extensive experiments demonstrate that our method outperforms prior works by a large margin, resulting in faithful reconstructions and impressive visual quality. Our method presents the first attempt to achieve high-quality 3D creation from a single image for general objects and enables various applications such as text-to-3D creation and texture editing.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d84616f108ccbd958735fef7622e58d148b32139

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “Furthermore, these methods require a time-intensive, object-specific optimization process to distill diffusion priors [7, 33, 40, 50, 60], combined with monocular cues (depth or normals [14, 38]), into neural 3D representations [43, 58].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “In order to achieve high-fidelity 3D creation, Make-It-3D [265] optimizes a neural radiance field by incorporating constraints from the reference image at the frontal view and diffusion prior at novel views, enhancing the coarse model into textured point clouds and increasing realism with diffusion ”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Text-to-3D methods can also be adapted for image-to-3D generation (Xu et al., 2023a; Tang et al., 2023b; Melas-Kyriazi et al., 2023) using image captioning models (Li et al., 2022; 2023a).”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, background): “As previously mentioned, several recent works leverage 2D diffusion models to perform per-shape optimization, allowing for the text-to-3D task [55, 35, 25] given that diffusion models are typically conditioned on text.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Subsequent studies have focused on enhancing 3D representations [6, 17, 42, 43], refining sampling schedules [13], and optimizing loss designs [46].”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “While some optimization-based approaches [32], [33], [34] directly train a 3D representation under the supervision of text-to-image (T2I) diffusion models [31], they require scene-specific optimization, which compromising their generalization capabilities.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Many works [43, 50, 52, 61, 66, 82] also apply such a distillation pipeline in the single-view reconstruction task.”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Following works have been further improving the 3D representations (Lin et al., 2023a; Tsalicoglou et al., 2023; Tang et al., 2023a; Chen et al., 2023), sampling schedules (Huang et al., 2023), and loss design (Wang et al., 2023c).”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Using text-based priors for single-image-to-3D has also shown promise [29, 30, 31], but requires a complex balancing of the image observation with additional constraints.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “For the task of text-to-3D [2, 5, 6, 23, 31, 52, 53, 61, 67, 69, 73, 82] or image-to-3D synthesis [42, 48, 50, 54, 58, 71], these methods typically optimize a 3D representation (i.e., NeRF, mesh, or SDF), and then leverage neural rendering to generate 2D images from various viewpoints.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “Recently, advances in image generation, such as DALL-E (Ramesh et al., 2021) and Stable Diffusion (Rombach et al., 2022), have inspired research that leverages the remarkable generalization capability of 2D diffusion models to enable multi-view supervision (Liu et al., 2023b; Tang et al., 2023).”