Text-based editing of talking-head video

By Ohad Fried, Ayush Kumar Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B. Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala. 2019

291 citations recorded by the source corpus.

Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

The Creation and Detection of Deepfakes (2020, methodology): “, see Refs [25], [55], [71], [73], [89], [104], [151], and [173]).”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, methodology): “On the other hand, 3D graphics modeling has been introduced in GAN-based methods due to its stability (Fried et al. 2019; Kim et al. 2018; Yi et al. 2020).”
Creating, Using, Misusing, and Detecting Deep Fakes (2022, methodology): “A related text-based video editing technique permits the addition, removal, and alteration of words in a video from, for example “she sells seashells by the sea shore” to “she sells ice cream by the sea shore” (Fried et al. 2019).”
Deep learning for deepfakes creation and detection: A survey (2019, unknown): “Audio-to-Video (A2V) [68] and Text-to-Video (T2V) [142].”
Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, methodology): “It surpasses the quality and speed of stateof-the-art neural rendering methods [11,33].”
HyperNeRF (2021, methodology): “2019] or conventional renderings [Fried et al. 2019; Kim et al. 2018; Martin-Brualla et al. 2018; Meshry et al. 2019] as a harness with which neural rendering can then be guided.”
Leveraging Frequency Analysis for Deep Fake Image Recognition (2020, background): “GANs produce sample outputs—images, audio signals, or complete video sequences—that are astonishingly effective at fooling humans into believing their veracity (Fried et al., 2019; Karras et al., 2019; Kumar et al., 2019; Song et al., 2020).”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “[16] utilize dynamic programming algorithm to reduce the training data to 1 hour.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “They analyze the performance by creating deepfakes using Audio-to-Video (A2V) [37] and Text-to-Video (T2V) [112] synthesis techniques.”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, methodology): “proposed in [186] a method that takes as input a video of a person speaking and the desired text to be spoken, and synthesises a new video in which the persons”
Deepfakes: Trick or treat? (2020, background): “Relatedly, Stanford researchers demonstrated text-to-speech (TTS) editing of a talking-head video by changing Apple’s price per share in a video announcement simply by substituting a number in the text transcript (Fried et al., 2019).”
Media Forensics and DeepFakes: An Overview (2020, background): “These include content generation [22], [23], image/video synthesis from semantic labels [24], [25] or sketches [26] or text [27], changes of style and attributes [22], [28], [29], domain translation [30], up to expression transfer [25], face swapping typical of deepfakes [31] and talking-head video ”
DurIAN: Duration Informed Attention Network For Multimodal Synthesis (2019, methodology): “During the reconstruction process, the 3D avatar facial movement can be reconstructed [27, 28] using the fhw32b 3D facial feature predicted using DurIAN model.”