Voice Separation with an Unknown Number of Multiple Speakers

By Eliya Nachmani, Yossi Adi, Lior Wolf. 2020

189 citations recorded by the source corpus.

We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.

Subfields: Artificial Intelligence, Pharmacy, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Pharmacy

In Speech & Audio

In Signal Processing

Citations in the corpus

High Fidelity Neural Audio Compression (2022, methodology): “For objective metrics, we use ViSQOL (Hines et al., 2012; Chinen et al., 2020) 2 , together with the Scale-Invariant Signal-to-Noise Ration (SI-SNR) (Luo & Mesgarani, 2019; Nachmani et al., 2020; Chazan et al., 2021).”
Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “Specifically, we compared our model to i) [22] which trained a separate model to separate a different number of speakers, denoted as Ensemble; ii) [7, 21] which trains one model to separate the maximum number of speakers, denoted as MaxOut.”
Real Time Speech Enhancement in the Waveform Domain (2020, methodology): “Additionally, deep learning based methods have also shown to be superior over traditional methods for the related task of a single-channel source separation [9, 10, 11].”
Music Source Separation in the Waveform Domain (2019, background): “We also compare to the more recent variant of DPRNN by Nachmani et al. [2020], Meta-Tasnet [Samuel et al., 2020], and Tasnet trained on extra data [Lancaster and Souviraà-Labastie, 2020].”
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, background): “Some neural-network-based speech separation methods [19, 39, 40, 41, 42] limit the number of outputs by their network architecture, and thus, there is no way to deal with the mismatch.”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “Hence, previous studies directly predict spectrogram [36,38] or waveform [7,37,39] as ˆ x a = f (˜ x a , x v ) and optimize the model with regression and adversarial losses.”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “This includes monaural speech separation methods, which address the problem of separating a mixture of speakers from a single audio track [23, 36, 60, 66], as well as monaural speech enhancement methods, which tackle the problem of removing nonspeech background sounds [8, 30, 42, 55, 56, 63] and rev”
Wavesplit: End-to-End Speech Separation by Speaker Clustering (2020, unknown): “This is particularly remarkable as Wavesplit is trained on only 1s long windows, unlike ConvTasNet and Dual-Path RNN that are both trained on 4s long windows.”
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals (2020, background): “4 Voice Separation [27] WSJ0-Nmix × multiple 20.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Speech separation is a special case of sound source separation where the goal is to separate the speech signal of a target speaker from background interference, including nonspeech noise [75, 42] and/or interfering speech from other speakers [79, 48].”
D3Net: Densely connected multidilated DenseNet for music source separation (2020, unknown): “* [20] 6.”
Attention Is All You Need In Speech Separation (2020, unknown): “The big advantage of SepFormer over RNN-based systems like [17,20,22] is the possibility to parallelize the computations over different time steps.”
Socio-Technological Challenges and Opportunities: Paths Forward (2021, unknown): “Individuals with hearing issues can use their Apple AirPods as basic hearing aids [Haselton, 2019], and people with average hearing can use AI-enabled voice assistants to eliminate background noise and focus on the conversation at hand [Nachmani et al., 2020].”