Neural 3D Video Synthesis from Multi-view Video

By Tianye Li, Miroslava Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven J. Lovegrove, Michael Goesele, Richard A. Newcombe and 1 more. Computer Vision and Pattern Recognition, 2021

473 citations recorded by the source corpus.

We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance field that represents scene dynamics using a set of compact latent codes. We are able to significantly boost the training speed and perceptual quality of the generated imagery by a novel hierarchical training scheme in combination with ray importance sampling. Our learned representation is highly compact and able to represent a 10 second 30 FPS multi-view video recording by 18 cameras with a model size of only 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for complex and dynamic scenes. We perform an extensive qualitative and quantitative evaluation that shows that our approach outperforms the state of the art. Project website: https://neural-3d-video.github.io/.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7b696063e4c8918a9bcea58efcc3d2507a91bdd4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Wenming Zhou
2. Yuqing Zhang
3. Xiaogang Jin
4. Xinyu Gao
5. Ziyi Yang
6. Shaohui Jiao

In Computer Vision and Pattern Recognition

1. Wenming Zhou
2. Hongye Yang
3. Yuqing Zhang
4. Xiaogang Jin
5. Li Zhang
6. Xinyu Gao
7. Xiatian Zhu
8. Zeyu Yang

Citations in the corpus

• NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, unknown): “For dynamic scene modeling, many works [56, 40, 45, 41, 28, 62, 26] propose to disentangle a 4D scene into a shared canonical space and a deformable field per frame.”
• 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, unknown): “Visualization of DyNeRF’s rendering results through our model, as referenced in [15]. progress and we will explore higher rendering quality on complex real scenes in the subsequent development.”
• HexPlane: A Fast Representation for Dynamic Scenes (2023, unknown): “For a comprehensive evaluation, we use two datasets with distinct settings: the high-resolution, multi-camera Plenoptic Video dataset [31], with challenging dynamic content and intricate visual details; the monocular D-NeRF dataset [58], featuring synthetic objects.”
• Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…Gaussians, as well as to adaptively control the density of the set of Gaussians. geometric prior, introducing 2D CNN to encode scene priors [22, 33], and leveraging the redundant information in multi-view videos [19] to set up keyframe compression storage, thereby accelerating the rendering speed.”
• Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (2023, methodology): “…line attempts to learn a 6D plenoptic function represented by a well-tailored implicit or explicit structure without direct modeling for the underlying motion to address this challenge (Li et al., 2022b; Fridovich-Keil et al., 2023; Cao & Johnson, 2023; Wang et al., 2023; Attal et al., 2023).”