One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

By Minghua Liu, Chao Xu, Haian Jin, Ling Chen, T. MukundVarma, Zexiang Xu, Hao Su. Neural Information Processing Systems, 2023

717 citations recorded by the source corpus.

Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c22ef963b2388cdbbfcc7a00b24f68710a7febd2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “One-2-3-45 [39] directly regresses SDFs from the output images of Zero123 and we use the official hugging face online demo [18] to produce the results.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Our optimization-based 2D lifting approach relies on 2D diffusion prior models (Liu et al., 2023b; Rombach et al., 2022), which may introduce unintended biases due to training data.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “We conduct the evaluation quality of geometry reconstruction from single-view images with other baselines, including Point-E [45], Shap-E [26] and One-2-3-45 [35].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “The baselines include SOTA single-image 3D generation methods: ShapE [36], One-2-3-45 [54], One-2-3-45++ [53], DreamGaus-sian [93], Wonder3D [58], TriplaneGaussian [114], and LGM [92].”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, methodology): “Diffusion Models have been employed for conditional Novel View Synthesis [3, 18, 19, 42].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “In addition, artiﬁcially constructing common sense or physics-compliant knowledge and applying them to generative models or simulators is a common strategy (e.g., ElastoGen [214], One-2-3-45 [215], PLoT [216]).”
A Survey on Generative Diffusion Models (2022, background): “Based on this work, [149] further extends Zero-1-to-3 by incorporating a pose estimation stage.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “One-2-3-45 [32] directly predict SDFs via SparseNeuS [39] by taking the generated multiple images of Zero123 [33].”
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model (2023, methodology): “Recent works like One-2-3-45 [11], SyncDreamer [13] and Consis-tent123 [10] build extra layers upon Zero-1-to-3 to obtain more 3D-consistent results.”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “Besides, using mesh representation can also bring convenience to applying additional post-processing steps to enhance the results, such as SDS optimization [3, 20] or texture baking [22].”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “Inspired by Zero123 [25], several recent works [20, 23, 24, 26, 28, 37, 44, 48, 62] finetune image or video diffusion models to generate novel view images as a stronger guidance for 3D generation.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, methodology): “For feed-forward approaches, we compare with One-2-3-45 [33] and Shap-E [25].”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “In cases where there is little uncertainty in what should appear at novel views, reasoning about generated views independently is sufficient for efficient 3D reconstruction [42].”
LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “In Figure 3, we compare LRM with One-2-3-45, a concurrent work to ours that achieves state-of-the-art single image to 3D reconstruction by generating multi-view images with 2D diffusion models (Liu et al., 2023a).”