Space-time Neural Irradiance Fields for Free-Viewpoint Video

By Wenqi Xian, Jia-Bin Huang, Johannes Peter Kopf, Changil Kim. Computer Vision and Pattern Recognition, 2020

548 citations recorded by the source corpus.

We present a method that learns a spatiotemporal neural irradiance field for dynamic scenes from a single video. Our learned representation enables free-viewpoint rendering of the input video. Our method builds upon recent advances in implicit representations. Learning a spatiotemporal irradiance field from a single video poses significant challenges because the video contains only one observation of the scene at any point in time. The 3D geometry of a scene can be legitimately represented in numerous ways since varying geometry (motion) can be explained with varying appearance and vice versa. We address this ambiguity by constraining the time-varying geometry of our dynamic scene representation using the scene depth estimated from video depth estimation methods, aggregating contents from individual frames into a single global representation. We provide an extensive quantitative evaluation and demonstrate compelling free-viewpoint rendering results.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/414da5e0716eb4707e033b7967d7671dfe71ab71

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HyperNeRF (2021, methodology): “2021], and modulation-based approaches directly condition the radiance field of the scene on some property of the input image and modify it accordingly [Li et al. 2021, 2020; Xian et al. 2020].”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, background): “NeRF has inspired significant follow-up work that has addressed some of its limitations, notably extensions to dynamic scenes [31, 7, 46], relighting [2, 3, 37], and incorporation of uncertainty [22].”
Dynamic View Synthesis from Dynamic Monocular Video (2021, methodology): “Very recently, several methods concurrently to ours have been proposed to extend NeRF for handling dynamic scenes [58, 27, 54, 43, 46].”
Editing Conditional Radiance Fields (2021, background): “Instead, several recent approaches implicitly encode a continuous volumetric representation of shape [14, 21, 38, 39, 42, 43, 48, 50, 54, 61] or both shape and view-dependent appearance [41, 45, 47, 59, 63, 66, 69, 76, 12, 73] in the weights of a neural network.”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “There exists also a large number of not-yet peer-reviewed, but impressive extensions of NeRF [Du et al. 2020; Gafni et al. 2020; Gao et al. 2020; Li et al. 2020; Martin-Brualla et al. 2020; Park et al. 2020; Rebain et al. 2020; Schwarz et al. 2020; Tretschk et al. 2020; Xian et al. 2020].”
Advances in Neural Rendering (2021, methodology): “To that end, some methods estimate a second static volume that is not conditioned on the deformation [LNSW21, GSKH21] or introduce soft regularization losses to constrain static scene content [XHKK21].”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “NeRF has recently been extended to dynamic scenes [16,31,47,48,51,64,71], though these approaches generally assume that motion is small.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, methodology): “Some methods [30, 20, 44, 14] extended NeRFs to dynamic scenes.”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for more practical scenarios beyond simple static scenes, including unbounded scenes [42,3], dynamic scenes [12,41], nonrigidly deforming objects [27,25,26,38], phototourism settings with changing ill”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “One line of research represents dynamic scenes by extending NeRF with an additional time dimension (T-NeRF) or additional latent code [16, 28, 29, 76].”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, background): “…introduce a set of sparse control points Here, p i denotes the learnable coordinate of control point i in the canonical space. o i is the learnable radius parameter of a radial-basis-function (RBF) kernel that controls how the impact of a control point on a Gaussian will decrease as their…”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, methodology): “For dynamic scene modeling, many works [56, 40, 45, 41, 28, 62, 26] propose to disentangle a 4D scene into a shared canonical space and a deformable field per frame.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “It has also been extended on many fronts, e.g . reﬂectance modeling for photorealistic relighting [3, 50] and dynamic scene modeling that integrates the motion of the world [25, 63, 39].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
NeuMan: Neural Human Radiance Field from a Single Video (2022, methodology): “As in Video-NeRF [49], simply minimizing Eq.”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “A common paradigm to model a dynamic scene is directly storing it in an MLP with 4D coordinates as the input [9, 44] or a 4D tensor with the time dimension or spatial dimensions compressed [1, 4, 7, 38].”
Nerfies: Deformable Neural Radiance Fields (2020, background): “Other concurrent works [29, 60] reconstruct space-time videos by recovering time-varying NeRFs while leveraging external supervision such as monocular depth estimation and flow-estimation to resolve ambiguities.”