Zero-1-to-3: Zero-shot One Image to 3D Object

By Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick. IEEE International Conference on Computer Vision, 2023

1,787 citations recorded by the source corpus.

We introduce Zero-1-to-3, a framework for changing the camera viewpoint of an object given just a single RGB image. To perform novel view synthesis in this under-constrained setting, we capitalize on the geometric priors that large-scale diffusion models learn about natural images. Our conditional diffusion model uses a synthetic dataset to learn controls of the relative camera viewpoint, which allow new images to be generated of the same object under a specified camera transformation. Even though it is trained on a synthetic dataset, our model retains a strong zero-shot generalization ability to out-of-distribution datasets as well as in-the-wild images, including impressionist paintings. Our viewpoint-conditioned diffusion approach can further be used for the task of 3D reconstruction from a single image. Qualitative and quantitative experiments show that our method significantly outperforms state-of-the-art single-view 3D reconstruction and novel view synthesis models by leveraging Internet-scale pre-training.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2c70684973bc4d7b6f8404a647b8031c4d3c8383

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, background): “Utilizing 3D-aware diffusion models [35, 67, 90] may be able to solve this problem.”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “We also include CLIP scores [73] and FID [28], which are two common metrics to evaluate image similarity in generation tasks [54,55,106].”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, methodology): “Recently, Zero123 [15] introduced an image-conditioned diffusion model which generates novel object views and is trained on Objaverse 1.0.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “Recently, advances in image generation, such as DALL-E (Ramesh et al., 2021) and Stable Diffusion (Rombach et al., 2022), have inspired research that leverages the remarkable generalization capability of 2D diffusion models to enable multi-view supervision (Liu et al., 2023b; Tang et al., 2023).”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, background): “Zero123 [23] demonstrates that Stable Diffusion can be fine-tuned to synthesize novel views by conditioning on relative camera poses.”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, neutral): “However, since Stable Diffusion is trained on a dataset with a noticeable forward-facing viewpoint bias [55], these training-free methods are limited and often suffer from severe performance issues, such as the Janus problem and multi-view inconsistency, which result in textures with significant…”
Continuous 3D Perception Model with Persistent State (2025, neutral): “With rapid advances in image and video generative models [57, 61, 62], much of the current research in 3D generation focuses on transferring priors from images and videos to 3D [45, 48, 56, 91].”
LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, background): “This stimulates the rapid developments of 3D content generation approaches [5, 14, 16, 21–24, 28, 30, 33, 34, 40, 45].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “…D REAM LLM’s context consistency feature for geometry-preserving tasks, including 3D content creation (Poole et al., 2023; Qi et al., 2023b; Liu et al., 2023d), representation learning (Dong et al., 2023; Qi et al., 2023a; Zhang et al., 2023a;f), scene comprehension (Zhang et al., 2023b;…”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “We found that embedding the camera pose as a low-dimensional vector (as in [36]) works well for in-domain samples, but generalizes poorly compared to raymap conditioning (see Section 3.1).”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “The application of Score Distillation Sampling to an image-conditioned diffusion model has been explored in recent work [14, 6, 33].”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “Nevertheless, we believe our proposed VSD and other contributions are orthogonal to these problems, and the generation quality can be further improved by introducing more techniques, such as using a more powerful text-to-image diffusion model that understands view-dependent prompts better [33, 40], ”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Liu et al. (2023) propose to fine-tune a pre-trained image variation diffusion model (sdv) on an extensive 3D render dataset for novel view synthesis.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “For this, Zero123 [25] repurposes an image diffusion model to a novel view synthesizer, conditioned on a single-view image and the pose difference be-tween the input and target views.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “We present the picked best results of Zero123 [33] after multiple runs.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Thereby, in image prompt cases, in contrast to previous image-conditioned approaches like Zero123 [19], which attempt to recover object 3D geometry based on image camera coordination system, Im-ageDream adopts canonical/world camera coordination as in MVDream.”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “In con-strast to GAN [4], some new works show that diffusion models can be employed in some 3D tasks, such as optical flow estimation [10,54], view synthesis [31,53,58], depth estimation [21,24,81], and normal estimation [32,35,45].”