Generative Novel View Synthesis with 3D-Aware Diffusion Models

By Eric Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, J. Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein. IEEE International Conference on Computer Vision, 2023

328 citations recorded by the source corpus.

We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our model samples from the distribution of possible renderings consistent with the input and, even in the presence of ambiguity, is capable of rendering diverse and plausible novel views. To achieve this, our method makes use of existing 2D diffusion backbones but, crucially, incorporates geometry priors in the form of a 3D feature volume. This latent feature field captures the distribution over possible scene representations and improves our method’s ability to generate view-consistent novel renderings. In addition to generating novel views, our method has the ability to autoregressively synthesize 3D-consistent sequences. We demonstrate state-of-the-art results on synthetic renderings and room-scale scenes; we also show compelling results for challenging, real-world objects.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/cf923fb70bbad20c485cef355444a08096747f68

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, methodology): “Since we are proposing a deterministic reconstruction method, we do not compare to methods that employ Score Distil-lation [47] or feed-forward diffusion models [3, 42].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, neutral): “Beyond the specific case of ego-exo, cross-view methods are explored for translation [130, 131, 134, 157], novel view synthesis [19, 90, 135, 137, 164, 168, 171], and aerial to ground matching [86, 132].”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, positive): “Subsequent advancements include Zhou et al.’s [53] extension to latent space with an epipolar feature transformer and Chan et al.’s [4] approach to enhance view consistency.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “However, these models are either category-specific [35], [38] or limited to handling toy-level objects with simple backgrounds.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, background): “Some works [6, 15, 26] try to address the multi-view consistency by jointly reasoning a 3D representation during NVS, but this comes at the cost of high computational and data requirements, often still resulting in unsatisfactory re-sults with inconsistent geometric and texture details.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “To generate consistent multi-view images, some efforts [3, 11, 18, 30, 34, 57, 59, 60, 62, 68, 70, 72, 74, 77, 81] are made to extend 2D diffusion models from single-view images to multi-view images.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “To improve the multiview consistency, these methods allow the diffusion model to condition on the input, previously generated images [40, 72, 78, 86, 91] or renderings from a neural field [5, 25, 70].”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “The recent surge in image generation using diffusion models [54, 56] enabled approaches to “imagine” images from novel viewpoints by incorporating pose transformation conditions [36] or novel view rendering feature maps [5, 82, 83].”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, background): “In novel view synthesis, recent single-image 3D diffusion models trained on real-world data rescale 3D scenes according to heuristics on depth statistics and condition their encoders on scene scale [4, 38, 51].”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “To overcome this limitation, several approaches train or fine-tune generative models with explicit image and pose conditioning [7, 36, 37, 38, 39, 40, 41].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, methodology): “Recently, some methods [2, 27, 65, 88] have employed diffusion models to aid sparse view reconstruction tasks.”
MVDream: Multi-view Diffusion for 3D Generation (2023, background): “Chan et al. (2023) enhanced the view consistency by re-projecting the latent features during diffusion denoising.”