Deep clustering: Discriminative embeddings for segmentation and separation

By John R. Hershey, Zhuo Chen, Jonathan Le Roux, Shinji Watanabe. 2015

1,455 citations recorded by the source corpus.

We address the problem of "cocktail-party" source separation in a deep learning framework called deep clustering. Previous deep network approaches to separation have shown promising performance in scenarios with a fixed number of sources, each belonging to a distinct signal class, such as speech and noise. However, for arbitrary source classes and number, "class-based" methods are not suitable. Instead, we train a deep network to assign contrastive embedding vectors to each time-frequency region of the spectrogram in order to implicitly predict the segmentation labels of the target spectrogram from the input mixtures. This yields a deep network-based analogue to spectral clustering, in that the embeddings form a low-rank pair-wise affinity matrix that approximates the ideal affinity matrix, while enabling much faster performance. At test time, the clustering step "decodes" the segmentation implicit in the embeddings by optimizing K-means with respect to the unknown assignments. Preliminary experiments on single-channel mixtures from multiple speakers show that a speaker-independent model trained on two-speaker mixtures can improve signal quality for mixtures of held-out speakers by an average of 6dB. More dramatically, the same model does surprisingly well with three-speaker mixtures.

Subfields: Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

Citations in the corpus

Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “Over the years, many attempts have been made to tackle this separation problem considering single microphone [1, 2], multiple microphones [3, 4], supervised and unsupervised learning [5, 6].”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “For research purposes only, we use the WSJ0-2mix and WSJ0-3mix datasets (Hershey et al., 2016) and we further expand the WSJ-mix dataset to four and five speakers and introduce WSJ0-4mix and WSJ0-5mix datasets.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “Our formulation of the problem resembles recent audio-visual and audio-only separation work [34,36,67,42].”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “Theproposed uPIT technique is algorithmically simpler yet performs on par with DPCL [36], [39] and comparable to DANets [37], [40], both of which involve separate embedding and clustering stages during inference.”
No Fuss Distance Metric Learning Using Proxies (2017, methodology): “It has successfully been employed for image retrieval [14], near duplicate detection [19], clustering [4] and zero-shot learning [8].”
Looking to listen at the cocktail party (2018, neutral): “In addition, a fundamental problem with audio-only speech separation is the label permutation problem [Hershey et al. 2016]: there is no easy way to associate each separated audio source with its corresponding speaker in the video [Hershey et al. 2016; Yu et al. 2017].”
SDR – Half-baked or Well Done? (2018, methodology): “Both SI-SDR and BSS eval’s SDR have recently been used by various studies [6–9, 11, 21–23, 25, 26] in the context of single-channel speaker-independent speech separation on the wsj0-2mix dataset [6], some of these studies reporting both figures [21–23, 25].”
Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, methodology): “From the figure we can see c1early that the validation MSE hardly decreases with the conventional approach due to the label permutation problem discussed in [8, 23].”
The Sound of Pixels (2018, background): “Sound source separation methods enable applications ranging from music/vocal separation [24], to speech separation and enhancement [25,26,27].”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, methodology): “The separation-only experiment is conducted on the widely-used WSJ0-2mix dataset [24].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “The inventions of deep clustering [1, 2, 3], deep attractor networks [4, 5] and permutation free training [1, 2, 6, 7] have dramatically improved the performance of single-channel speaker-independent multispeaker speech separation, demonstrating overwhelming advantages over previous methods includin”
AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario (2021, methodology): “WSJ0-2mix [13], and its variants WHAM! [14] and WHAMR! [15] are mostly adopted for speech separation evaluation, which consists of artificially mixed speech samples using Wall Street Journal dataset.”
Speaker-Aware Neural Network Based Beamformer for Speaker Extraction in Speech Mixtures (2017, background): “The problem can be also alleviated by using long segments, though this disables the possibility of realtime processing and can hurt the performance of the methods [11, 12].”
ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis (2020, background): “It is often applied to images, and has found numerous applications such as visual products retrieval [16,23,1], person re-identification [37,30], face recognition [22], few-shot learning [28,14], and clustering [11].”
Supervised Speech Separation Based on Deep Learning: An Overview (2017, positive): “1055 were the first to address speaker-independent multi-talker sepa- 1056 ration in the DNN framework [69].”
FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks (2019, methodology): “1 Dataset and neural network We evaluated our system on two-speaker speech separation problem using WSJ0-2mix dataset [8, 11], which contains 30 hours of training and 10 hours of validation data.”
Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues (2019, methodology): “Most such studies focus on blind source separation (BSS) approaches such as deep clustering [1] or permutation invariant training [2], which separates an observed speech mixture into each of its sources without any prior information (i.”