You said that?

By Joon Son Chung, Amir Jamaludin, Andrew Zisserman. 2017

276 citations recorded by the source corpus.

We present a method for generating a video of a talking face. The method takes as inputs: (i) still images of the target face, and (ii) an audio speech segment; and outputs a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we propose an encoder-decoder CNN model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on tens of hours of unlabelled videos. We also show results of re-dubbing videos using speech from a different person.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, background): “The dataset is audio-visual, so is also useful for a number of other applications, for example – visual speech synthesis [9, 10], speech separation [11, 12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13, 14] and training face…”
Deep Audio-visual Learning: A Survey (2020, methodology): “…alignment Kumar et al. [94] and Supasorn et al. [95] Generated keypoints by a time-delayed LSTM Needed retraining for another identity Chung et al. [96] Developed an encoder-decoder CNN model suitable for more identities Jalalifar et al. [97] Combined RNN and GAN and applied keypoints For a…”
Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, methodology): “7 as well as in the video, we show generated images of stateof-the-art image-based methods [5,36].”
Realistic Speech-Driven Facial Animation with GANs (2019, methodology): “There is no publicly available implementation for the Speech2Vid method proposed in [10] but a pre-trained model is provided, which we can use for comparison.”
Learning Individual Styles of Conversational Gesture (2019, background): “[12] generate an image of a talking face from a still image of the speaker and an input speech segment by learning a joint embedding of the face and audio.”
Self-supervised learning of a facial attribute embedding from video (2018, background): “…not make use of the temporal ordering information inherent in a video; nor do we predict future frames conditioned on a number of past frames [44], or explicitly predict the motion between frames as a convolutional kernel [22, 58], or condition the generation on another modality (e.g. voice [8]).”
Media Forensics and DeepFakes: An Overview (2020, background): “animated based on an audio input sequence [46], [47].”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “A number of previous studies [14, 72, 45, 44] leverage skip connections for better input identity preserving.”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, background): “Chung et al. [2017]; Song et al. [2019]; Vougioukas et al. [2019] generated full natural human face images with GANs or encoder-decoder CNNs. [Pham et al. 2018] estimated blendshape parameters.”
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos (2019, methodology): “Instead of using RGB videos as was done in previous work [30, 14], Averbuch et al. and Chung et al. used ID-type photos [4, 9], which are easily obtained on social networks.”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Earlier methods try to synthesize the lip motions that fulfill the training data of a still facial image [5, 13, 8, 42, 7, 37].”
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos (2018, background): “Nowadays, the only thing needed to create a forged facial image is simply a short video of the target person [1, 2] or an ID photo [3, 4].”
Fighting Fake News: Image Splice Detection via Learned Self-Consistency (2018, background): “With only rudimentary editing skills, it is now possible to create realistic image composites [4, 5], ﬁll in large image regions [1, 6, 7], generate plausible video from speech [8, 9], etc.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, methodology): “Nonetheless, all the recent photo-realistic talking face generation methods [3, 12, 27, 1, 28, 35] failed to consider this problem.”
Lip Movements Generation at a Glance (2018, methodology): “In terms of LMD and PSNR, our full model shows better performance than methods that use discriminator [17] or reconstruction loss [10].”
The Conversation: Deep Audio-Visual Speech Enhancement (2018, methodology): “The enhancement method proposed here is complementary to other examples of audio-visual learning, such as visual speech synthesis [27, 28] and lip reading [29, 30, 31], which has also been shown to improve ASR performance in noisy environments [32, 33].”