Novel View Synthesis with Diffusion Models

By Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi. International Conference on Learning Representations, 2022

348 citations recorded by the source corpus.

We present 3DiM, a diffusion model for 3D novel view synthesis, which is able to translate a single input view into consistent and sharp completions across many views. The core component of 3DiM is a pose-conditional image-to-image diffusion model, which takes a source view and its pose as inputs, and generates a novel view for a target pose as output. 3DiM can generate multiple views that are 3D consistent using a novel technique called stochastic conditioning. The output views are generated autoregressively, and during the generation of each novel view, one selects a random conditioning view from the set of available views at each denoising step. We demonstrate that stochastic conditioning significantly improves the 3D consistency of a naive sampler for an image-to-image diffusion model, which involves conditioning on a single fixed view. We compare 3DiM to prior work on the SRN ShapeNet dataset, demonstrating that 3DiM's generated completions from a single view achieve much higher fidelity, while being approximately 3D consistent. We also introduce a new evaluation methodology, 3D consistency scoring, to measure the 3D consistency of a generated object by training a neural field on the model's output views. 3DiM is geometry free, does not rely on hyper-networks or test-time optimization for novel view synthesis, and allows a single model to easily scale to a large number of scenes.

View Paper: https://www.semanticscholar.org/paper/23a1b0fdb857fbbe328ffe254df33ff615acc5ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Watson et al.[47] pioneered diffusion model applications in view synthesis as the pipeline in Sitzmann et al.[39] using the ShapeNet dataset.”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “For instance, Watson et al. (2023) first applied diffusion models to view synthesis using the ShapeNet dataset (Sitzmann et al., 2019).”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “To demonstrate the effectiveness of our autoregressive synthesis method, which aggregates conditioning feature volumes from autoregressively selected generated images, we compare to an adaptation of the stochastic conditioning method proposed in 3DiM [28].”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, methodology): “With the recent success of diffusion models for image generation [20, 51], researchers have applied diffusion models to the task of novel view synthesis — modeling the distribution of unseen views given observations from known views [15, 29, 63].”
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, methodology): “In [47] a diffusion model has been used to generate multiple views of a given input image.”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, background): “Recently, the authors of [46] have proposed to directly generate multiple 2D views of an object, which can then be reconstructed in 3D using a NeRF-like model.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, methodology): “Input coordinates are concatenated with sinusoidal embeddings following the work of Mildenhall et al. [38] and Watson et al. [67].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Beyond the specific case of ego-exo, cross-view methods are explored for translation [130, 131, 134, 157], novel view synthesis [19, 90, 135, 137, 164, 168, 171], and aerial to ground matching [86, 132].”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “[68] proposes to synthesize novel views with a pose-conditioned 2D diffusion model, yet the results are not intrinsically 3D.”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “Early promising diffusion-based results [2, 16, 49, 62, 101, 116] have mainly been restricted by lacking availability of useful real-world multi-view training data.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “To generate consistent multi-view images, some efforts [3, 11, 18, 30, 34, 57, 59, 60, 62, 68, 70, 72, 74, 77, 81] are made to extend 2D diffusion models from single-view images to multi-view images.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “To overcome this limitation, several approaches train or fine-tune generative models with explicit image and pose conditioning [7, 36, 37, 38, 39, 40, 41].”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, unknown): “…and Nichol, 2022, Ding et al., 2021], image-to-image generation [Saharia et al., 2022c,a, Whang et al., 2022], 3D synthesis [Poole et al., 2022, Watson et al., 2022], text-to-speech [Chen et al., 2021a, Kong et al., 2021, Chen et al., 2021b], and density estimation [Kingma et al., 2021, Song…”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “Recently, approaches like Zero-1-to-3 [46] and 3Dim [91] adapted a 2D diffusion model [71] to become view-dependent and utilized this view-dependent diffusion as a 3D prior.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, methodology): “Prior work [57] adopted a similar pipeline but did not demonstrate zero-shot generalization capability.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, background): “However, these models are either category-specific [35], [38] or limited to handling toy-level objects with simple backgrounds.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “To improve the multiview consistency, these methods allow the diffusion model to condition on the input, previously generated images [40, 72, 78, 86, 91] or renderings from a neural field [5, 25, 70].”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Watson et al. (2022) employ an image-to-image diffusion model to synthesize novel views of an object conditioned on a single view, allowing many consistent views to be synthesized autoregressively.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “With recent rapid progress of diffusion models, 3DiM [54] introduces a pose-conditional diffusion model that generates a novel view conditioned on a source view and a target pose.”