Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction

By G. Gafni, Justus Thies, Michael Zollhöfer, Matthias Nießner. Computer Vision and Pattern Recognition, 2020

660 citations recorded by the source corpus.

We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face1. Digitally modeling and reconstructing a talking human is a key building-block for a variety of applications. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoint or headposes is required. In contrast to state-of-the-art approaches that model the geometry and material properties explicitly, or are purely image-based, we introduce an implicit representation of the head based on scene representation networks. To handle the dynamics of the face, we combine our scene representation network with a low-dimensional morphable model which provides explicit control over pose and expressions. We use volumetric rendering to generate images from this hybrid representation and demonstrate that such a dynamic neural scene representation can be learned from monocular input data only, without the need of a specialized capture setup. In our experiments, we show that this learned volumetric representation allows for photorealistic image generation that surpasses the quality of state-of-the-art video-based reenactment methods.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/890398bb6364141f8b4a798fbc1c1605a871bd1d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Benefiting from these advantages, NeRF has been widely used in many fields, such as 3D modeling [53, 58], human face/body digitization [12, 17, 42, 43, 48, 55], generating 4D free-view video [31,39], etc.”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for generative modeling [8, 36], dynamic scenes [23, 33], nonrigidly deforming objects [13, 34], phototourism settings with changing illumination and occluders [26, 43], and reflectance modeling for r”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 51, 71], deformation [13, 38, 40, 41, 56], self-calibration [19, 27, 28, 35, 62], meta-learning [53], dynamic scene modeling [14, 25, 33, 42, 65], and generative modeling [5, 22, 45].”
Advances in Neural Rendering (2021, methodology): “NeRFace [GTZN21] is the ﬁrst approach that uses a morphable model to implicitly control a neural radiance ﬁeld (see Figure 10).”
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields (2021, background): “The success of NeRF has also inspired many follow-up works that extend the NeRF to dynamic scenes [30, 33, 8, 40], relighting [3, 39], generative models [37, 25, 4, 13], etc.”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, methodology): “The closest work to ours is NerFace [18], which recovers an animatable neural radiance field from a single monocular video by conditioning on the expression parameters from a 3DMM.”
HyperNeRF (2021, background): “…some canonical radiance field [Park et al. 2020; Pumarola et al. 2020; Tretschk et al. 2021], and modulation-based approaches directly condition the radiance field of the scene on some property of the input image and modify it accordingly [Gafni et al. 2021; Li et al. 2021, 2020; Xian et al. 2020].”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “[16], since we use the corresponding pixel on the background image as the last sample for each ray, the implicit networks learn to predict low density values for the foreground samples if the ray is passing through a background pixel, and high density values for foreground pixels.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “Some concurrent works also use neural scene representations for reﬁning camera motion [75], video reenactment [50], and facial models [15, 16].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “Concurrent to Nerfies was NerFace [82] (December 2020), which also used per-frame learned latent”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, positive): “NeRF has inspired many subsequent works, which extend its neural volumetric scene representation to application domains including dynamic and deformable scenes [24], avatar animation [10, 25], and even phototourism [19].”
Mixture of volumetric primitives for efficient neural rendering (2021, background): “There exists also a large number of not-yet peer-reviewed, but impressive extensions of NeRF [Du et al. 2020; Gafni et al. 2020; Gao et al. 2020; Li et al. 2020; Martin-Brualla et al. 2020; Park et al. 2020; Rebain et al. 2020; Schwarz et al. 2020; Tretschk et al. 2020; Xian et al. 2020].”
FastNeRF: High-Fidelity Neural Rendering at 200FPS (2021, background): “Finally, note that while we train this proof-of-concept method on a dataset captured using multiple cameras, the approach can be extended to use only a single camera, similarly to [7].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “This approach has been applied for deforming faces [12, 14, 45], moving human bodies [43, 53, 56], and objects [10,29,42,44,59].”
Baking Neural Radiance Fields for Real-Time View Synthesis (2021, positive): “The NeRF representation has been remarkably successful for view synthesis, and follow-on works have extended NeRF for generative modeling [6, 35], dynamic scenes [22, 31], non-rigidly deforming objects [13, 32], and relighting [2, 38].”
Neural 3D Video Synthesis from Multi-view Video (2021, neutral): “[16] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.”