The Conversation: Deep Audio-Visual Speech Enhancement

By Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman. 2018

397 citations recorded by the source corpus.

Our goal is to isolate individual speakers from multi-talker simultaneous speech in videos. Existing works in this area have focussed on trying to separate utterances from known speakers in controlled environments. In this paper, we propose a deep audio-visual speech enhancement network that is able to separate a speaker's voice given lip regions in the corresponding video, by predicting both the magnitude and the phase of the target signal. The method is applicable to speakers unheard and unseen during training, and for unconstrained environments. We demonstrate strong quantitative and qualitative results, isolating extremely challenging real-world examples.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, background): “The dataset is audio-visual, so is also useful for a number of other applications, for example – visual speech synthesis [9, 10], speech separation [11, 12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13, 14] and training face…”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “8× as manys videos), as in [47], and modeled ambient sounds by sampling background audio tracks from AudioSet approximately 8% of the time.”
Looking to listen at the cocktail party (2018, neutral): “The threshold was set empirically using synthetic mixtures of clean speech and non-speech interfering noise at different, known SNR”
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020, methodology): “Using the LRW test set, we create 28K pairs, and this set measures the performance on frontal/near-frontal videos [2].”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “The audio waveform is extracted from the video following the methods of [6, 72].”
Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues (2019, methodology): “On the other hand, SpeakerBeam-AV-MTL could achieve better performance than Baseline-A and Baseline-V (see Table 1) even in such a situation, while maintaining the performance with both audio and visual clues (Clues = AV).”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, result): “This distinguishes our approach from studies that directly predict unbounded or unconstrained phase differences [22], [23], clean phases [24] and real and imaginary components of target sources [25], [26], or fully complex neural network approaches [27].”
Deep Audio-visual Learning: A Survey (2020, methodology): “Gabbay et al. [5] Predict speaker’s voice based on faces in video used as a ﬁlter Can only be used in controlled environments Afouras et al. [6] Generate a soft mask for ﬁltering in the wild Requires training a separate model for each speaker of interest Speaker Separation Lu et al. [20]…”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, neutral): “It consists of a 3D convolutional layer (3D Conv) followed by a ResNet18 block [2].”
Voxceleb: Large-scale speaker verification in the wild (2020, background): “…also useful for a number of other applications, for example, visual speech synthesis (Chung et al., 2017; Karras et al., 2017), speech separation (Afouras et al., 2018; Ephrat et al., 2018), cross-modal transfer from face to voice or vice versa (Nagrani et al., 2018b; 2018a), emotion recognition…”
2.5D Visual Sound (2018, background): “We add a sigmoid layer after the upconvolution layers to bound the complex mask values to [1, 1], similar to [10].”
Co-Separating Sounds of Visual Objects (2019, background): “Recent methods leverage deep learning for separating speech [8, 31, 3, 11], musical instruments [52, 13, 51], and other objects [12].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “This ensures the groundtruth of the separated speech is known for quantitative evaluation, following standard practice [2, 19].”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “In the speech domain, these approaches rely on facial recognition [6, 18, 70] and/or lip motion [1, 18, 39] to suppress sounds that do not correspond to the speaker in the visual stream.”
Neural Target Speech Extraction: An overview (2023, methodology): “Furthermore, the flexibility of NNs to integrate different modalities combined with the high modeling capability of face recognition or lip-reading systems offered new possibilities for speaker-open visual clue-based TSE [7], [8].”
Large-Scale Visual Speech Recognition (2018, background): “Other related advances include works using vision for silent speech reconstruction [43–46] and for separating an audio signal into its individual speech sources [47, 48].”
Learning Words by Drawing Images (2019, background): “Recently proposed models that learn these correspondences can be used to perform tasks such as visually-guided audio source separation, or localizing source of a sound within an image or video [35, 44, 42, 17, 1, 18, 5].”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “We follow previous work [2, 19] and use a mix-and-separate learning procedure.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, methodology): “Recent efforts using heuristics to mine web video with imputed labels [1, 12, 37] are similarly limited.”