AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

By Yudong Guo, Keyu Chen, Sen Liang, Yongjin Liu, Hujun Bao, Juyong Zhang. IEEE International Conference on Computer Vision, 2021

527 citations recorded by the source corpus.

Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a high-fidelity talking-head video corresponding to the audio signal is synthesized using volume rendering. Another advantage of our framework is that not only the head (with hair) region is synthesized as previous methods did, but also the upper body is generated via two individual neural radiance fields. Experimental results demonstrate that our novel framework can (1) produce high-fidelity and natural results, and (2) support free adjustment of audio signals, viewing directions, and background images. Code is available at https://github.com/YudongGuo/AD-NeRF.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a81ef3138365f762334736769174738413cccdd9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “They mostly focused on multi-modal NeRFs such as [13], [14], only citing these two and the original NeRF paper [1] in their survey, as well as four more papers [15] [16][17][18] in their supplementary materials.”
Advances in Neural Rendering (2021, background): “Guo et al. [GCL ∗ 21] propose an audio driven neural radiance ﬁeld (AD-NeRF) which is inspired by NeRFace.”