Dynamic View Synthesis from Dynamic Monocular Video

By Chen Gao, Ayush Saraf, Johannes Peter Kopf, Jia-Bin Huang. IEEE International Conference on Computer Vision, 2021

545 citations recorded by the source corpus.

We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match the input video). To resolve the ambiguity, we introduce regularization losses to encourage a more physically plausible solution. We show extensive quantitative and qualitative results of dynamic view synthesis from casually captured videos.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6b0a81d050e0bc0e49010c49f51ec3fcbaab6b28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “NeRF has recently been extended to dynamic scenes [16,31,47,48,51,64,71], though these approaches generally assume that motion is small.”
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (2023, background): “The two most common schemes are (1) modeling a deformation field on top of a static canonical field [8, 9, 17, 27, 30, 36, 43], or (2) directly learning a radiance field conditioned on time [12, 16, 17, 28, 41].”
Revealing Occlusions with 4D Neural Fields (2022, methodology): “Our model uses a continuous representation similar to methods in neural rendering and computer graphics [16,17, 40, 43, 63], which also enjoy significant computational advantages from the compact scene representation.”
NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior (2022, background): “Previous works [9, 17, 47] simply take mono-depth into a depth-wise loss along with NeRF training.”
Advances in Neural Rendering (2021, methodology): “To that end, some methods estimate a second static volume that is not conditioned on the deformation [LNSW21, GSKH21] or introduce soft regularization losses to constrain static scene content [XHKK21].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, methodology): “Some methods [7, 18, 35] represent dynamic scenes as 4D radiance fields but face extensive computational costs due to ray point sampling and volume rendering.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “Local editing of NeRFs ideally requires an efﬁcient, open-set method for coordinate-level decomposition.”
Neural 3D Video Synthesis from Multi-view Video (2021, unknown): “[17] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “A common paradigm to model a dynamic scene is directly storing it in an MLP with 4D coordinates as the input [9, 44] or a 4D tensor with the time dimension or spatial dimensions compressed [1, 4, 7, 38].”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Some concurrent works also use neural scene representations for refining camera motion [75], video reenactment [50], and facial models [15, 16].”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “In most cases [6, 10, 23, 23, 26], positional encoding [35] is used to expand information.”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “…inception, various works have been proposed to enhance performance in diverse scenarios, such as different resolutions of reconstruction [2, 3], the reduced number of training samples [32, 36, 42, 44, 47], and reconstruction of large realistic scenes [29, 41] and dynamic scenes [12, 23, 24, 33].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “One stream of dynamic NeRF methods [Du et al. 2021; Gao et al. 2021; Li et al. 2021a; Xian et al. 2021] model deformations in scenes by extending radiance fields with an additional time dimension.”
Urban Radiance Fields (2021, background): “Some research has been directed towards us-ing NeRF-like models for 3D reconstruction and novel view synthesis from images captured in natural environments (“in situ”) [19, 35, 60, 69, 75].”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “One line of research represents dynamic scenes by extending NeRF with an additional time dimension (T-NeRF) or additional latent code [17, 31, 32, 81].”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, unknown): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”