D-NeRF: Neural Radiance Fields for Dynamic Scenes

By Albert Pumarola, Enric Corona, Gerard Pons-Moll, Francesc Moreno-Noguer. Computer Vision and Pattern Recognition, 2020

2,021 citations recorded by the source corpus.

Neural rendering techniques combining machine learning with geometric reasoning have arisen as one of the most promising approaches for synthesizing novel views of a scene from a sparse set of images. Among these, stands out the Neural radiance fields (NeRF) [31], which trains a deep network to map 5D input coordinates (representing spatial location and viewing direction) into a volume density and view-dependent emitted radiance. However, despite achieving an unprecedented level of photorealism on the generated images, NeRF is only applicable to static scenes, where the same spatial location can be queried from different images. In this paper we introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a single camera moving around the scene. For this purpose we consider time as an additional input to the system, and split the learning process in two main stages: one that encodes the scene into a canonical space and another that maps this canonical representation into the deformed scene at a particular time. Both mappings are simultaneously learned using fully-connected networks. Once the networks are trained, D-NeRF can render novel images, controlling both the camera view and the time variable, and thus, the object movement. We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions. Code, model weights and the dynamic scenes dataset will be available at [1].

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/694bdf6e5906992dad2987a3cc8d1a176de691c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics (2023, background): “For example, both Pumarola et al. [30] and Park et al. [27] decompose time-dependent neural fields into an inverse displacement field and canonical time-invariant neural fields.”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “These methods either learn a static canonical radiance field with deformation [54, 56, 68] or a dynamic radiance field directly conditioned on time [31, 15].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “A KL divergence term is added to minimize the divergence of the ray’s distribution and this noisy depth distribution (See [22] for details).”
HyperNeRF (2021, methodology): “…into two distinct categories: deformation-based approaches apply a spatially-varying deformation to some canonical radiance field [Park et al. 2020; Pumarola et al. 2020; Tretschk et al. 2021], and modulation-based approaches directly condition the radiance field of the scene on some property of…”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, methodology): “The deformation field is represented as translational vector field [51] or SE(3) field [46].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for more practical scenarios beyond simple static scenes, including unbounded scenes [42,3], dynamic scenes [12,41], nonrigidly deforming objects [27,25,26,38], phototourism settings with changing ill”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Neural rendering with implicit representations has become a widely-used technique for solving many vision and graphics tasks ranging from view synthesis [5, 15, 25], to re-lighting [12,13], to pose and shape estimation [17,21,31], to 3D-aware image synthesis and editing [3, 11, 23], to modeling dyna”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “While this approach is in principle applicable to non-rigid objects, this remains to be tested, and extensions based on recent work on non-rigid scenes in 3D reconstruction and novel view synthesis [9, 18, 26, 28, 31, 43] might be necessary.”
Advances in Neural Rendering (2021, methodology): “D-NeRF [PCPMMN21] uses an unregularized ray-bending MLP to model deformations of a single or multiple synthetic objects segmented from the background and observed by virtual cameras.”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, methodology): “For monocular evaluation, we use the synthetic dataset provided by D-NeRF [35] and select 3 scenes (“lego”, “standup” and “jumpingjacks”) from this dataset, with the numbers of training frames ranging from 50 to 200.”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “…[1,4,5,10,11,15,21,23,45, 47,48,57,65,73] or use posed images to jointly optimize geometry and appearance [6,7,18,25,29,33,36–39, 44, 49, 50, 54, 55 Triplanesareefficient, scalingwiththesurfacearearatherthan volume, and naturally integrate with expressive, fine-tuned 2D generator architectures.”