Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

By Edgar Tretschk, Ayush Kumar Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, Christian Theobalt. IEEE International Conference on Computer Vision, 2020

637 citations recorded by the source corpus.

We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a ‘bullet-time’ video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision. Our formulation enables dense correspondence estimation across views and time, and compelling video editing applications such as motion exaggeration. Our code will be open sourced.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eaee62ae051a9cd8065db7f5a444a3209d31e989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “NeRF has recently been extended to dynamic scenes [16,31,47,48,51,64,71], though these approaches generally assume that motion is small.”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “[12, 43, 46, 57] explicitly model dynamic scenes by a warp field or velocity field to deform a canonical radiance field.”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, methodology): “For dynamic scene modeling, many works [56, 40, 45, 41, 28, 62, 26] propose to disentangle a 4D scene into a shared canonical space and a deformable field per frame.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, neutral): “K-Planes [28], HexPlane [12] and Tensor4D [77] factorize 4D spacetime domain into 2D feature planes for compact model size.”
HyperNeRF (2021, methodology): “…deformation-based approaches apply a spatially-varying deformation to some canonical radiance field [Park et al. 2020; Pumarola et al. 2020; Tretschk et al. 2021], and modulation-based approaches directly condition the radiance field of the scene on some property of the input image and…”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, methodology): “Meanwhile, some works [48, 67] consider using NeRF to model dynamic scenes and using Multi-Layer Perceptron (MLP) to predict scene changes.”
SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes (2023, background): “…we introduce a set of sparse control points Here, p i denotes the learnable coordinate of control point i in the canonical space. o i is the learnable radius parameter of a radial-basis-function (RBF) kernel that controls how the impact of a control point on a Gaussian will decrease as their…”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for more practical scenarios beyond simple static scenes, including unbounded scenes [42,3], dynamic scenes [12,41], nonrigidly deforming objects [27,25,26,38], phototourism settings with changing ill”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, background): “…seek to enhance the quality of dynamic neural rendering from various aspects, including segmenting static and dynamic objects in the scene [39, 42], incorporating depth information [1] to introduce and time γ ( t ) with positional encoding as input to a deformation MLP network to obtain the…”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, background): “This is an additional neural network that first maps observations in deformed space to observations in a canonical space, where the implicit function is then evaluated [32, 34, 34, 38, 42, 51, 53].”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 51, 71], deformation [13, 38, 40, 41, 56], self-calibration [19, 27, 28, 35, 62], meta-learning [53], dynamic scene modeling [14, 25, 33, 42, 65], and generative modeling [5, 22, 45].”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The success of NeRF has also inspired many follow-up works that extend the NeRF to dynamic scenes [30, 33, 8, 40], relighting [3, 39], generative models [37, 25, 4, 13], etc.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “Another line of research employs individual MLPs to represent a deformation field and a canonical field [11, 49, 50, 54, 70, 83], where the canonical field depicts a static scene, and the deformation field learns coordinate maps to the canonical space over time.”