VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking

By Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John R. Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, Ignacio Lopez-Moreno. 2018

428 citations recorded by the source corpus.

In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Shinji Watanabe
4. Yossi Adi
5. Zhuo Chen
6. Lior Wolf
7. Li Deng
8. Yoshua Bengio

In Computer Vision and Pattern Recognition

1. Dong Yu
2. Yossi Adi
3. Luc Van Gool
4. Lior Wolf
5. Dacheng Tao
6. Alexei A. Efros
7. Neil Zeghidour
8. Xiaogang Wang

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Shinji Watanabe
4. Yossi Adi
5. Zhuo Chen
6. Lior Wolf
7. Takuya Yoshioka
8. Eliya Nachmani

Citations in the corpus

• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…et al., 2023e) semantic token, speaker prompt speech Speech Enhancement (SE) ♣ (Wang et al., 2023b) noisy speech speech Target Speech Extraction (TSE) ♣ (Wang et al., 2018) mixed speech, speaker prompt speech Singing Voice Synthesis (SVS) (Liu et al., 2022) phoneme (with duration), speaker…”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “Inthe contextoftransformingnon-cleanspeech,anotherparticularly usefultaskistargetspeakerextraction[22],[23],[24].”
• Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Wang et al. (2018c) propose a novel method to separate speech signals in the time-frequency domain.”
• Wavesplit: End-to-End Speech Separation by Speaker Clustering (2020, positive): “The separation stack is classical and resembles previous architectures conditioned on pre-trained speaker vectors (Wang et al., 2018a), or trained with PIT (Luo & Mes-garani, 2019).”
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “In general, all these approaches show that speaker embeddings, when extracted from an available additional speech utterance from the target speaker, can be useful, confirming the results obtained in the AO domain [262].”
• SpeechPainter: Text-conditioned Speech Inpainting (2022, background): “Also relevant, but not directly related to our work, are methods that enhance speech based on modalities other than text: using facial crops of the target speaker, denoising [6], inpainting speech [15] or separating a target speaker from a mixture [4, 20], and enhancing speech based on accelerometer”
• SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures (2019, methodology): “The method introduced in [33] proposes concatenating a dvector [34] extracted from the adaptation utterance with one of”
• Personalized speech enhancement: new models and Comprehensive evaluation (2021, methodology): “Wang et al. proposed VoiceFilter [8, 9] to extract the target speaker for improving ASR accuracy.”