The Weizmann Institute of Science

By Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, Michael Rubinstein. 1964

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, methodology): “The dataset is audio-visual, so is also useful for a number of other applications, for example – visual speech synthesis [9, 10], speech separation [11, 12],”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “[46] proposed a method for separating the speech a user-chosen speaker, with a model trained on a dataset of inthe-wild educational videos.”
The Sound of Pixels (2018, background): “[16] shows how person appearance could help solving the cocktail party problem in speech domain.”
Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, background): “Recent work is solving this ’cocktail party’ issue using visual cues [7].”
Neural Target Speech Extraction: An overview (2023, methodology): “13 IEEE SIGNAL PROCESSING MAGAZINE | May 2023 | similar voices [8].”
Learning Robust and Multilingual Speech Representations (2020, methodology): “Unlabeled speech pretraining corpus For pretraining, we collected a diverse and noisy speech corpus from several existing datasets: the subset of Audio Set (Gemmeke et al., 2017) containing speech examples, the audio part of AVSpeech (Ephrat et al., 2018), and the Common Voice (CV)2 dataset in all 2”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “xt, vision can be combined with speech as well. Such tasks include audio-visual speech recognition [50]–[52], speaker recognition [53]–[55], as well as speech diarisation [56], [57], separation [58], [59] and enhancement [60], which mostly focused on the use of visual features to improve the robustn”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, methodology): “[55]), because a priori knowledge of the number of speakers is needed to choose a specific trained multi-speaker model.”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “proposed a rectangular coordinate-wise cRM made with sigmoid compressions onto each of the real and imaginary parts of the output of the model (Ephrat et al., 2018).”
Weighted Speech Distortion Losses for Neural-Network-Based Real-Time Speech Enhancement (2020, background): “Several DNN-based approaches report high-quality enhancement using generous look-ahead [8, 9], but their performance for decreasing lookahead is not well investigated.”
Co-Separating Sounds of Visual Objects (2019, methodology): “We find spectrogram masks to work better than direct prediction of spectrograms or raw waveforms for source separation, confirming reports in [49, 8, 13].”
VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking (2018, methodology): “While training the VoiceFilter system, the input audios are divided into segments of 3 seconds each and are converted, if necessary, to single channel audios with a sampling rate of 16 kHz.”
The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results (2020, methodology): “DNN based methods are shown to outperform conventional SE techniques in suppressing nonstationary noises [7]–[10].”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “The result has enabled major advances in a range of downstream supervised audiovisual tasks [45, 35, 17].”
WavThruVec: Latent speech representation as intermediate features for neural speech synthesis (2022, methodology): “We therefore use AVSpeech [28], a large-scale audio-visual dataset comprising speech video clips with no interfering background noises, initially intended for speech separation and audio-visual event localization.”
Grounding Object Detections With Transcriptions (2019, background): “More concretely, (Zhao et al., 2018; Ephrat et al., 2018) ground sound to image regions and use it a posteriori to perform source sound localitzation and isolation.”
Audio-Visual Model Distillation Using Acoustic Images (2019, background): “More recent works [2, 3, 9, 31, 38] train both visual and audio networks aiming at learning multimodal representations useful for many applications, such as cross-modal retrieval, speech separation, sound source localization, action recognition, and on/off-screen audio source separation.”