Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model

By Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, Sai Bi. International Conference on Learning Representations, 2023

433 citations recorded by the source corpus.

Text-to-3D with diffusion models has achieved remarkable progress in recent years. However, existing methods either rely on score distillation-based optimization which suffer from slow inference, low diversity and Janus problems, or are feed-forward methods that generate low-quality results due to the scarcity of 3D training data. In this paper, we propose Instant3D, a novel method that generates high-quality and diverse 3D assets from text prompts in a feed-forward manner. We adopt a two-stage paradigm, which first generates a sparse set of four structured and consistent views from text in one shot with a fine-tuned 2D text-to-image diffusion model, and then directly regresses the NeRF from the generated images with a novel transformer-based sparse-view reconstructor. Through extensive experiments, we demonstrate that our method can generate diverse 3D assets of high visual quality within 20 seconds, which is two orders of magnitude faster than previous optimization-based methods that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3af134a559a618b3185390646d49d1d4e7ffab45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “…2022), to introduce semantics and multi-view guidance for image-to-3D reconstruction (Liu et al., 2023b; Tang et al., 2023; Deng et al., 2023; Shen et al., 2023b; Anciukeviˇcius et al., 2023; Melas-Kyriazi et al., 2023; Metzer et al., 2023; Xu et al., 2023; Qian et al., 2023; Li et al., 2023a).”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “Concurrently, Instant3D [19] proposes a diagram that predicts 3D shapes via an enhanced LRM with multi-view input generated by diffusion models.”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, methodology): “Our method also uses 3D Gaussians as an underlying representation but predicts them from as few as a single image.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “We show that pairing these models with 3D reconstruction, similar to [17, 58], enables efficient and high quality 3D creation.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, methodology): “Our method adopts a multi-view reconstruction setting similar to Instant3D [19].”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, neutral): “Three text categories are”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “Following [46], we filter 100 k high-quality objects, and render 32 images at random viewpoints with a fixed 50 ◦ field of view under ambient lighting.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “Some other works based on LRM architecture are proposed for multi-view image generation [76], sparse-view reconstruction [31], and pose-free sparse-view reconstruction [64].”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “We empirically find that the perceptual loss in [14] based on VGG-19 network [55] provides a more stable training than LPIPS [75] used in [27,32,63,68], and we use it in this work.”