Wavesplit: End-to-End Speech Separation by Speaker Clustering

By Neil Zeghidour, David Grangier. 2020

295 citations recorded by the source corpus.

We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2/3mix), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “Namely, we include in our evaluation iTDCN++ [17] and Wavesplit [33].”
Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “In this work, we focus on fully supervised voice separation using a single microphone, which has seen a great leap in performance following the recent success of deep learning models considering both frequency domain [1, 2, 7, 8, 9, 10], and time-domain [11, 12, 13, 14, 15, 16] modeling.”
Voice Separation with an Unknown Number of Multiple Speakers (2020, background): “Zeghidour & Grangier (2020) propose to infer a set of speaker representations via clustering to better separate the speakers.”
Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization (2021, methodology): “In other methods, the number-of-speaker-wise output branches is trained independently, and the most probable is used during inference [41].”
Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation (2020, methodology): “To overcome this problem, time-domain approach is proposed in paper [14] , which directly model the mixture waveform using an encodedecoder framework and has made great progress in recent years [15, 16, 17, 18, 19, 20, 21].”
Attention Is All You Need In Speech Separation (2020, unknown): “We explored the use of dynamic mixing (DM) data augmentation [23] which consists in on-the-fly creation of new mixtures from single speaker sources.”
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, methodology): “Even if the method itself is designed not to limit the number of speakers, there is rarely experimental evidence to show how the model actually works under mismatched conditions [43, 44].”
LibriMix: An Open-Source Dataset for Generalizable Speech Separation (2020, methodology): “Finally, models relying on some kind of speaker identity representation [2,6,15] cannot easily detect overﬁtting, since wsj0-2mix’s speakers are shared between the training and validation sets.”