Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss

By Lele Chen, Ross K. Maddox, Zhiyao Duan, Chenliang Xu. Computer Vision and Pattern Recognition, 2019

473 citations recorded by the source corpus.

We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions. Instead of learning a direct mapping from audio to video frames, we propose first to transfer audio to high-level structure, i.e., the facial landmarks, and then to generate video frames conditioned on the landmarks. Compared to a direct audio-to-image approach, our cascade approach avoids fitting spurious correlations between audiovisual signals that are irrelevant to the speech content. We, humans, are sensitive to temporal discontinuities and subtle artifacts in video. To avoid those pixel jittering problems and to enforce the network to focus on audiovisual-correlated regions, we propose a novel dynamically adjustable pixel-wise loss with an attention mechanism. Furthermore, to generate a sharper image with well-synchronized facial movements, we propose a novel regression-based discriminator structure, which considers sequence-level information along with frame-level information. Thoughtful experiments on several datasets and real-world samples demonstrate significantly better results obtained by our method than the state-of-the-art methods in both quantitative and qualitative comparisons.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/a0852cd9a026bc90168fa85fa422cb0e48f98394

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, background): “There is a number of techniques for audio-driven facial reenactment [2,3,5,9,35,36] but only a few generate photo-realistic, natural, full frame images [27].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “On the other hand, subject-agnostic models [4,9,11,20,22,24,30,56,60,68,70,81,82,86,92,93,100] only need a single image of the target person, who is not seen during training, to synthesize arbitrary motions.”
Deep Audio-visual Learning: A Survey (2020, background): “[101] Dynamic pixelwise loss Required multistage”
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, methodology): “Thus, previous facial landmark-based methods [2,50] and 2D flow-based audio to expression networks [37,38] may generate the distorted face since the head motion and expression are not fully disentan-gled in their representation.”
The Creation and Detection of Deepfakes (2020, background): “[27] ATVGnet:”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “As stated before, VoxCeleb2 [9] is one of the largest in-the-wild face video datasets, and it contains massive face videos, that have contributed to the development of many fields [13,89,16,85,5,88,86,30].”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, background): “Then we training the network with the metric learning step using ArcLoss, which leads LRS3-TED VoxCeleb2 Real Chen [5] Baseline Wang [17] Wiles [18] Zakharov [20] Real Chen [5] Baseline Wang [17] Wiles [18] Zakharov [20] Top-1 72.62% 1.99% 3.85% 2.23% 1.93% 1.77% 42.46% 2.40% 3.04% 1.64% 1.87% 1.99%”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, unknown): “photorealistic talking-head animation [12, 15, 42, 52, 63, 67, 69].”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “There are a branch of talking head generation methods [5, 13, 8, 42, 7, 37] entirely lying in the image domain.”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, methodology): “ in degrees) and head position (again normalized by the width of the face). The measure indicates head pose differences, like nods and tilts. 8 • Zhou et al. GT GT (cropped) [Vougioukas et al. 2019] [Chen et al. 2019] Ours Ours (cropped) Single c artoon image Generated samples Artistic painting 2D c”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, background): “lso have an additional overhead of requiring clean training data of each target speaker to generate for that speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] a”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, methodology): “The pose information is inherently coupled with facial movements, which affects both reconstruction-based [14, 72] and 2D landmark-based methods [10, 18].”
Regeneration Learning: A Learning Paradigm for Data Generation (2023, background): “…are some spurious correlations between X and Y , e.g., the speaking timber in source speech has no correlation with the head pose in target video (Chen et al. 2019) for talking-head video synthesis, and some information in target melody has no correlation with source lyric in lyric-to-melody…”