ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

By Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, Yonghong Tian. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

378 citations recorded by the source corpus.

Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose ViewCrafter, a novel method for synthesizing high-fidelity novel views from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with significantly improved camera pose control accuracy. To further enlarge the generation range of novel views, we tailored a progressive view synthesis strategy to expand the point cloud and the areas covered by the novel views, which can be further integrated with a camera trajectory planning algorithm to automatically reveal and address occlusions in different scenes. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity novel views. Our project webpage and code are available at https://drexubery.github.io/ViewCrafter/.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/bde2a93e64558fae1b447e6beec0d3624dfc790a

Citations in the corpus