SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

By Vikram S. Voleti, C. Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, Varun Jampani. European Conference on Computer Vision, 2024

411 citations recorded by the source corpus.

We present Stable Video 3D (SV3D) -- a latent video diffusion model for high-resolution, image-to-multi-view generation of orbital videos around a 3D object. Recent work on 3D generation propose techniques to adapt 2D generative models for novel view synthesis (NVS) and 3D optimization. However, these methods have several disadvantages due to either limited views or inconsistent NVS, thereby affecting the performance of 3D object generation. In this work, we propose SV3D that adapts image-to-video diffusion model for novel multi-view synthesis and 3D generation, thereby leveraging the generalization and multi-view consistency of the video models, while further adding explicit camera control for NVS. We also propose improved 3D optimization techniques to use SV3D and its NVS outputs for image-to-3D generation. Extensive experimental results on multiple datasets with 2D and 3D metrics as well as user study demonstrate SV3D's state-of-the-art performance on NVS as well as 3D reconstruction compared to prior works.

View Paper: https://www.semanticscholar.org/paper/03e2cfa44b64489fb98f09dfbd940043fbef90ad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Ying Shan
2. Xintao Wang
3. Weihao Cheng
4. Jiale Xu
5. Yiming Gao
6. Shenghua Gao
7. Kelvin C. K. Chan
8. Jinjin Gu

Citations in the corpus

• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Therefore, unlike SVD that linearly increases the guidance scale, we employ a triangular classifier-free guidance scheme [118] to permit genuine long-horizon rollouts.”
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “…our sparse-view reconstruction model accepts free-viewpoint images as input, so we can integrate arbitrary multi-view generation model into our framework, e.g ., MVDream [43], ImageDream [51], SyncDreamer [24], SPAD [17] and SV3D [48], to achieve both text-to-3D and image-to-3D assets creation.”