One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

By Ting-Chun Wang, Arun Mallya, Ming-Yu Liu. Computer Vision and Pattern Recognition, 2020

653 citations recorded by the source corpus.

We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person’s appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/85089ce8830e9fcd24ee6b157c22bfa34d7d828a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, neutral): “Conversely, explicit methods warp the source image to the target by either 2D optical flow [22, 26–28, 46, 52], 3D deformation field [3, 17, 36], or directly sawpping the face of target image [18].”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “GAN based methods [35, 40, 44, 45] commonly employ warping functions to spatially transform the reference image according to input motion for generating sequential video frames.”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “The usage of audio features instead of regressed expression coefﬁ-cients [43] or facial landmarks [49] is beneﬁcial for alleviating the training cost of intermediate translation network and preventing potential semantic mismatching issue between audio and visual signals.”
Thin-Plate Spline Motion Model for Image Animation (2022, background): “1) transfers the motion of the object in the driving video to the static object in the source image, which is widely used for video conferencing [31], movie effects [21] and entertainment videos.”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Examples include various image and video editing tasks, image inpainting [13, 47, 46, 41], local image editing [42, 2], low bit-rate video conferencing [40], image super resolution [28, 12], image colorization [4, 27], and extracting 3D models [30].”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Speciﬁcally, certain works [61, 47, 59] learn the warping between pixels without deﬁned structural information.”