HeadNeRF: A Realtime NeRF-based Parametric Head Model

By Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, Juyong Zhang. Computer Vision and Pattern Recognition, 2021

331 citations recorded by the source corpus.

In this paper, we propose HeadNeRF, a novel NeRF-based parametric head model that integrates the neural radiance field to the parametric representation of the human head. It can render high fidelity head images in real-time on modern GPUs, and supports directly controlling the generated images' rendering pose and various semantic attributes. Different from existing related parametric models, we use the neural radiance fields as a novel 3D proxy instead of the traditional 3D textured mesh, which makes that HeadNeRF is able to generate high fidelity images. However, the computationally expensive rendering process of the original NeRF hinders the construction of the parametric NeRF model. To address this issue, we adopt the strategy of integrating 2D neural rendering to the rendering process of NeRF and design novel loss terms. As a result, the rendering speed of HeadNeRF can be significantly accelerated, and the rendering time of one frame is reduced from 5s to 25ms. The well designed loss terms also improve the rendering accuracy, and the fine-level details of the human head, such as the gaps between teeth, wrinkles, and beards, can be represented and synthesized by HeadNeRF. Extensive experimental results and several applications demonstrate its effectiveness. The trained parametric model is available at https://github.com/CrisHY1995/headnerf.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c63c28feb43fdfd938e17e707bba06823fb9b7ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Advances in Neural Rendering (2021, methodology): “Besides these subject-speciﬁc training methods, Head-Nerf [HPX ∗ 21] and MoFaNeRF [ZZSC21] propose a generalized model to represent faces under different views, expressions and illumination.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Human NeRF Face Nerfies [39], HyperNeRF [40], HeadNeRF [139]”
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, methodology): “Since we predict the realistic 3D facial coefficients, our method can also be used in other modalities directly, i.e ., personalized 2D visual dubbing [43], 2D Cartoon animation [50], 3D face animation [8,44] and NeRF-based 4D talking-head generation [15].”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Current tasks, such as novel view synthesis [32,25,61,12,52] and 3d generation [9,61,8,26,21,11], are trained on in-the-wild image datasets [42,44] which lacks facial dynamics to provide natural geometries.”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, methodology): “HeadNeRF [11] learns a NeRF-based parametric head model with 2D neural rendering for efficiency.”