VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

By Ruohan Gao, Kristen Grauman. 2021

261 citations recorded by the source corpus.

We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous back-ground sounds and/or other human speakers. Whereas existing methods focus on learning the alignment between the speaker’s lip movements and the sounds they generate, we propose to leverage the speaker’s face appearance as an additional prior to isolate the corresponding vocal qualities they are likely to produce. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement, and generalizes well to challenging real-world videos of diverse scenarios. Our video results and code: http://vision.cs.utexas.edu/projects/VisualVoice/.

Subfields: Cognitive Neuroscience, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Meanwhile, visual information is increasingly used in historically audio-only tasks like speech transcription, voice recognition, audio spatialization [5,80,104,161], speaker diarization [10,83], and source separation [57,78,82].”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “For these two subtasks, masking-based methods (in magnitude spectrogram domain [16, 41], complex spectrogram domain [14, 19, 55], or on learned basis [33, 34]) are widely adopted where the enhancement model f predicts a mask M̂ = f(x̃a, xv).”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Extracting the shared information between visual and audio representations could lead to the speech content space by synchronizing both the modalities, which is also proven to be beneﬁcial for various downstream audio-visual tasks [17, 40, 73, 36, 25].”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, methodology): “By integrating facial frames of a target speaker with a noisy audio spectrogram, for example, recent deep learning models can generate a mask for the spectrogram that suppresses irrelevant voices and background sounds from the output [1,11,18].”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “As stated before, VoxCeleb2 [14] is one of the largest in-the-wild face video datasets, and it contains massive face videos, that have contributed to the development of many fields [19,103,23,98,10,102,99,38].”
Separate Anything You Describe (2023, unknown): “1) Audio-Visual Sound Separation: In the computer vision community, there has been active research focusing on utilizing visual information to extract target sounds in speech [36], [37], music [12], and acoustic events [13].”