Neural Radiance Flow for 4D View Synthesis and Video Processing

By Yilun Du, Yinan Zhang, Hong-Xing Yu, Josh Tenenbaum, Jiajun Wu. IEEE International Conference on Computer Vision, 2020

345 citations recorded by the source corpus.

We present a method, Neural Radiance Flow (NeRFlow), to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene. By enforcing consistency across different modalities, our representation enables multi-view rendering in diverse dynamic scenes, including water pouring, robotic interaction, and real images, outperforming state-of-the-art methods for spatial-temporal view synthesis. Our approach works even when being provided only a single monocular real video. We further demonstrate that the learned representation can serve as an implicit scene prior, enabling video processing tasks such as image super-resolution and de-noising without any additional supervision.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d67720b062f5dde1de3298d3140b7bbd332003cd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Mixture of volumetric primitives for efficient neural rendering (2021, background): “There exists also a large number of not-yet peer-reviewed, but impressive extensions of NeRF [Du et al. 2020; Gafni et al. 2020; Gao et al. 2020; Li et al. 2020; Martin-Brualla et al. 2020; Park et al. 2020; Rebain et al. 2020; Schwarz et al. 2020; Tretschk et al. 2020; Xian et al. 2020].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “Our work belongs to the latter; while the other two works [31, 15] regularize the training primarily with flow information, ours does so with dynamic scene depth.”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “This approach has been applied for deforming faces [12, 14, 45], moving human bodies [43, 53, 56], and objects [10,29,42,44,59].”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “[12] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu.”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Others focus on moving human bodies [85, 58, 75] or more general objects [61, 37, 86, 57, 11, 36].”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “While this approach is in principle applicable to non-rigid objects, this remains to be tested, and extensions based on recent work on non-rigid scenes in 3D reconstruction and novel view synthesis [9, 18, 26, 28, 31, 43] might be necessary.”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “One stream of dynamic NeRF methods [Du et al. 2021; Gao et al. 2021; Li et al. 2021a; Xian et al. 2021] model deformations in scenes by extending radiance fields with an additional time dimension.”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “The seminal work, Neural Radiance Fields (NeRF) [21], has been widely studied and advanced in a variety of applications [2, 3, 32, 19, 23, 13, 25], including novel view synthesis [21, 18], 3D generation [25, 10], deformation [23, 26, 28], video [15, 35, 7, 24, 14].”
Revealing Occlusions with 4D Neural Fields (2022, methodology): “Our model uses a continuous representation similar to methods in neural rendering and computer graphics [16,17, 40, 43, 63], which also enjoy significant computational advantages from the compact scene representation.”
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, background): “(c) Methods that represent the 3D scene in a canonical timestep and use a deformation field to warp this to the rest of the timesteps [8, 21, 22, 28, 29, 32, 40, 44].”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “K-Planes [28], HexPlane [12] and Tensor4D [77] factorize 4D spacetime domain into 2D feature planes for compact model size.”
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (2023, background): “The two most common schemes are (1) modeling a deformation field on top of a static canonical field [8, 9, 17, 27, 30, 36, 43], or (2) directly learning a radiance field conditioned on time [12, 16, 17, 28, 41].”