Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

By Zhengqi Li, Simon Niklaus, Noah Snavely, Oliver Wang. Computer Vision and Pattern Recognition, 2020

1,038 citations recorded by the source corpus.

We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input. To do this, we introduce Neural Scene Flow Fields, a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for varieties of in-the-wild scenes, including thin structures, view-dependent effects, and complex degrees of motion. We conduct a number of experiments that demonstrate our approach significantly outperforms recent monocular view synthesis methods, and show qualitative results of space-time view synthesis on a variety of real-world videos.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/13034a395d5c6728c9b11e777828d9998018cbf6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for more practical scenarios beyond simple static scenes, including unbounded scenes [42,3], dynamic scenes [12,41], nonrigidly deforming objects [27,25,26,38], phototourism settings with changing ill”
MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo (2021, background): “Following works have tried to advance its performance on view synthesis [31, 27]; other relevant works extend it to support other neural rendering tasks like dynamic view synthesis [26, 36, 43], challenge scenes [29, 52], pose estimation [32], realtime rendering [53], relighting [37, 1, 9], and edit”
Point-NeRF: Point-based Neural Radiance Fields (2022, methodology): “However, most methods [3, 26, 38, 53] still follow the original NeRF framework and train per-scene MLPs to represent radiance fields.”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “…inception, various works have been proposed to enhance performance in diverse scenarios, such as different resolutions of reconstruction [2, 3], the reduced number of training samples [32, 36, 42, 44, 47], and reconstruction of large realistic scenes [29, 41] and dynamic scenes [12, 23, 24, 33].”
HyperNeRF (2021, methodology): “PSNR↑ MS-SSIM↑ LPIPS↓ Avg↓ NeRF [Mildenhall et al. 2020] 21.3 .745 .490 .126 NV [Lombardi et al. 2019] 25.3 .880 .214 .0630 NSFF [Li et al. 2020] 25.5 .863 .242 .0663 Nerfies [Park et al. 2020] 27.7 .908 .193 .0487 Nerfies (w/o elastic) 28.0 .909 .193 .0476 Hyper-NeRF (DS) 28.3 .914 .185 .0456…”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, methodology): “This allows for accelerating existing approaches for: reconstruction of dynamic scenes [30, 34, 14], singleimage reconstruction [39], quality improvement [35, 47] and others [28].”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, methodology): “Some methods [30, 20, 44, 14] extended NeRFs to dynamic scenes.”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “There exists also a large number of not-yet peer-reviewed, but impressive extensions of NeRF [Du et al. 2020; Gafni et al. 2020; Gao et al. 2020; Li et al. 2020; Martin-Brualla et al. 2020; Park et al. 2020; Rebain et al. 2020; Schwarz et al. 2020; Tretschk et al. 2020; Xian et al. 2020].”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “While this approach is in principle applicable to non-rigid objects, this remains to be tested, and extensions based on recent work on non-rigid scenes in 3D reconstruction and novel view synthesis [9, 18, 26, 28, 31, 43] might be necessary.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “Note that the default hyper-parameters for NSFF [ ?”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “A series of subsequent NeRF works [Li et al. 2021a; Park et al. 2021a,b; Pumarola et al. 2021; Tretschk et al. 2021a] improve radiance field construction towards dynamic scenes.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “NeRF has recently been extended to dynamic scenes [16,31,47,48,51,64,71], though these approaches generally assume that motion is small.”
TensoRF: Tensorial Radiance Fields (2022, background): “This representation has been quickly extended and applied in diverse graphics and vision applications, including generative models [6,26], appearance acquisition [2,4], surface reconstruction [40,27], fast rendering [33,47,12], appearance editing [42,19], dynamic capture [17,28].”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for generative modeling [8, 36], dynamic scenes [23, 33], nonrigidly deforming objects [13, 34], phototourism settings with changing illumination and occluders [26, 43], and reflectance modeling for r”