Looking to listen at the cocktail party

By Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, Michael Rubinstein. 2018

377 citations recorded by the source corpus.

We present a joint audio-visual model for isolating a single speech signal from a mixture of sounds such as other speakers and background noise. Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video. In this paper, we present a deep network-based model that incorporates both visual and auditory signals to solve this task. The visual features are used to "focus" the audio on desired speakers in a scene and to improve the speech separation quality. To train our joint audio-visual model, we introduce AVSpeech, a new dataset comprised of thousands of hours of video segments from the Web. We demonstrate the applicability of our method to classic speech separation tasks, as well as real-world scenarios involving heated interviews, noisy bars, and screaming children, only requiring the user to specify the face of the person in the video whose speech they want to isolate. Our method shows clear advantage over state-of-the-art audio-only speech separation in cases of mixed speech. In addition, our model, which is speaker-independent (trained once, applicable to any speaker), produces better results than recent audio-visual speech separation methods that are speaker-dependent (require training a separate model for each speaker of interest).

Subfields: Cognitive Neuroscience, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Robust and Multilingual Speech Representations (2020, neutral): “Unlabeled speech pretraining corpus For pre-training, we collected a diverse and noisy speech corpus from several existing datasets: the sub-set of Audio Set (Gemmeke et al., 2017) containing speech examples, the audio part of AVSpeech (Ephrat et al., 2018), and the Common Voice (CV) 2 dataset in al”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, neutral): “Meanwhile, visual information is increasingly used in historically audio-only tasks like speech transcription, voice recognition, audio spatialization [5,80,104,161], speaker diarization [10,83], and source separation [57,78,82].”
SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “This approach was later extended by Looking2Listen [19], which uses visual information from facial crops segmented from videos to disentangle different speakers that talk simultaneously.”
End-to-End Neural Speech Coding for Real-Time Communications (2022, neutral): “We use a mean-square error on the power-law compressed spectrum between the original and the decoded signals for reconstruction loss [15].”
Audio-Visual Embodied Navigation (2019, neutral): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
Capture, Learning, and Synthesis of 3D Speaking Styles (2019, positive): “The relation between speech and facial motion has previously been used to separate audio-visual speech [22] and for audio-video driven facial animation [38].”
Listen to Look: Action Recognition by Previewing Audio (2019, neutral): “Recent work uses audio for an array of video understanding tasks outside of action recognition, including self-supervised representation learning [43, 5, 7, 41, 33, 54], audio-visual source separation [41, 2, 13, 19, 76, 21], localizing sounds in video frames [6, 48, 55], and generating sounds from ”
WHAM!: Extending Speech Separation to Noisy Environments (2019, neutral): “l that high SNR, intelligible speech be removed from the ambient noise corpus. To accomplish this, we used an inverted approach to the one used to remove overly noisy speech when creating AVSpeech in [7]. We ﬁrst process the ambient recordings with the commercially available iZotopeRX7DialogueIsolat”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, positive): “[40, 39, 13, 17, 32] used the visual-audio correspondence to separate sound sources.”
What Makes Multimodal Learning Better than Single (Provably) (2021, neutral): “For example, [35, 23, 22, 31, 20] combine RGB and depth images to improve semantic segmentation; [8, 16] fuse audio with video to do scene understanding; researchers also explore audio-visual source separation and localization [49, 11].”
Self-Supervised Speech Representation Learning: A Review (2022, neutral): “AVSpeech [210] is another large-scale audio-visual dataset used in SSL research, comprising 4.”
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (2018, positive): “Contemporarily, certain works study the relation between video and audio to achieve speech separation [13, 1].”
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, positive): “The inclusion of visual features has aided in simplifying object classification and source separation [7], [8], [9], but”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, positive): “As stated before, VoxCeleb2 [14] is one of the largest in-the-wild face video datasets, and it contains massive face videos, that have contributed to the development of many fields [19,103,23,98,10,102,99,38].”
Audio-Visual Event Localization in Unconstrained Videos (2018, neutral): “Concurrently, some interesting and related works on sound source separation, localization and audio-visual representation learning are explored in [13,16,40,47,59].”
AudioCaps: Generating Captions for Audios in The Wild (2019, neutral): “…et al., 2018; Senocak et al., 2018; Afouras et al., 2018; Gao et al., 2018; Arand-jelovic and Zisserman, 2018; Zhao et al., 2018), along with blind source separation aided by visual features (Ephrat et al., 2018) and learning of audio-visual multisensory representation (Owens and Efros, 2018).”
LRS3-TED: a large-scale dataset for visual speech recognition (2018, neutral): “TED videos have previously been used for audio-visual datasets for these reasons [9].”
Learning to Separate Object Sounds by Watching Unlabeled Video (2018, positive): “Concurrent with our work, other new methods for audio-visual source separation are being explored speciﬁcally for speech [1,23,28,58] or musical instruments [84].”