Nerfies: Deformable Neural Radiance Fields

By Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, S. M. Seitz, Ricardo Martin-Brualla. IEEE International Conference on Computer Vision, 2020

1,503 citations recorded by the source corpus.

We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0f1af3f94f4699cd70a554f68f8f9e2c8e3d53dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Specifically, we disable non-rigid motions at the beginning of optimization, and then bring them back in a coarse-to-fine manner [24,47].”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, methodology): “For dynamic scene modeling, many works [56, 40, 45, 41, 28, 62, 26] propose to disentangle a 4D scene into a shared canonical space and a deformable field per frame.”
Revealing Occlusions with 4D Neural Fields (2022, methodology): “Numerous efforts have been made to extend NeRF to dynamic scenes [16, 17, 40, 43, 63], but in addition to requiring per-scene retraining, occlusions are typically explicitly ignored by applying losses over the non-occluded scene only.”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “[12, 43, 46, 57] explicitly model dynamic scenes by a warp field or velocity field to deform a canonical radiance field.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, background): “tion and novel view synthesis given a set of posed camera images [3,38,44].”
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, background): “(c) Methods that represent the 3D scene in a canonical timestep and use a deformation field to warp this to the rest of the timesteps [8, 21, 22, 28, 29, 32, 40, 44].”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “The vast scale of animations available in OBJAVERSE can support a wide range of research in temporal 3D learning, such as building text-based animation generative models [76], representing object changes over time with NERFs [54,59], and temporal self-supervised learning via.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Mean-while, [22, 23, 26] have challenged the static hypothesis, expanding the boundary of novel view synthesis for dynamic scenes.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, background): “While originally proposed to reconstruct a static scene with a set of posed images, it has since been quickly extended to dynamic scenes [29,15,30] and uncalibrated scenes [45,17].”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, methodology): “Inspired by the progressive training strategy commonly used in prior NeRF methods [28], [51], [56], we implement a coarse-to-fine optimization strategy. begins by training on a subset of anchors representing lower LOD levels and progressively activates finer LOD levels throughout optimization,…”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, methodology): “5: Qualitative comparisons between NeRF[22], NeRF w/ WPE[25], Mip-NeRFfull[2], and BungeeNeRF.”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “This is an additional neural network that first maps observations in deformed space to observations in a canonical space, where the implicit function is then evaluated [32, 34, 34, 38, 42, 51, 53].”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “Conversely, disentangled methods [23, 30, 31, 34, 39] employ a deformation field to model a scene in canonical space by mapping point coordinates at a given time to this space.”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “A series of subsequent NeRF works [Li et al. 2021a; Park et al. 2021a,b; Pumarola et al. 2021; Tretschk et al. 2021a] improve radiance field construction towards dynamic scenes.”
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance (2023, background): “The former makes decomposing static and dynamic components easy [37, 40], but struggles with changes in scene topology (e.g. when a new object appears), while the latter makes disentangling static and dynamic objects hard.”