Single-Channel Multi-Speaker Separation Using Deep Clustering

By Yusuf Ziya Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, John R. Hershey. 2016

454 citations recorded by the source corpus.

Deep clustering is a recently introduced deep learning architecture that uses discriminatively trained embeddings as the basis for clustering. It was recently applied to spectrogram segmentation, resulting in impressive results on speaker-independent multi-speaker separation. In this paper we extend the baseline system with an end-to-end signal approximation objective that greatly improves performance on a challenging speech separation. We first significantly improve upon the baseline system performance by incorporating better regularization, larger temporal context, and a deeper architecture, culminating in an overall improvement in signal to distortion ratio (SDR) of 10.3 dB compared to the baseline of 6.0 dB for two-speaker separation, as well as a 7.1 dB SDR improvement for three-speaker separation. We then extend the model to incorporate an enhancement layer to refine the signal estimates, and perform end-to-end training through both the clustering and enhancement stages to maximize signal fidelity. We evaluate the results using automatic speech recognition. The new signal approximation objective, combined with end-to-end training, produces unprecedented performance, reducing the word error rate (WER) from 89.1% down to 30.8%. This represents a major advancement towards solving the cocktail party problem.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “, 2018), DPCL++ (Isik et al., 2016), CBLDNN-GAT (Li et al.”
Music Source Separation in the Waveform Domain (2019, methodology): “In the ﬁeld of monophonic speech source separation, spectrogram masking methods have enjoyed good performance [Kolbæk et al., 2017, Isik et al., 2016].”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “The proposed uPIT technique is algorithmically simpler yet performs on par with DPCL [36], [39] and comparable to DANets [37], [40], both of which involve separate embedding and clustering stages during inference.”
Looking to listen at the cocktail party (2018, neutral): “For such channels, most of the videos comprise a We discard segments for which the predicted speech SNR is below 17 dB (marked by the gray dotted line in the plot). single speaker, and both the video and audio are generally of high quality.”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, background): “To alleviate this issue, some robust ASR research focuses on mapping the out-of-domain data to in-domain data using enhancementbased methods [5, 6, 7], which generally requires parallel data from both domains.”
Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, methodology): “Hershey et aL [23, 24] made significant progress with their deep clustering (OPCL) technique.”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, neutral): “DPCL++ [31] 13.”
Single Channel Speech Separation with Constrained Utterance Level Permutation Invariant Training Using Grid LSTM (2018, methodology): “In this work, performance was evaluated with global normalized signal-to-distortion ratio (GNSDR, same as “SDR improvement” in [8, 9, 11, 12]) using the toolbox in [22].”
Deep Audio-visual Learning: A Survey (2020, background): “Some studies tried to solve the problem of audio separation with only the audio modality and achieved exciting results [14, 15] .”
New Era for Robust Speech Recognition (2017, methodology): “It is reported that the factored model in [25] outperforms the unfactored model in [24] in terms of word error rate (WER) on a voice search database with two microphones.”
The Phasebook: Building Complex Masks via Discrete Representations for Source Separation (2019, background): “The field of speech separation and speech enhancement has witnessed dramatic improvements in performance with the recent advent of deep learning-based techniques [1–11].”
Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, background): “Starting from deep clustering (DC) [1, 2] and permutation invariant training (PIT) [3, 4], a variety of separation models have been shown effective in separating overlapped speech [5, 6].”
Universal Sound Separation (2019, methodology): “Separation performance is measured using scaleinvariant signal-to-distortion ratio improvement (SI-SDRi) [7, 12], which evaluates the fidelity of a signal estimate ŝ, represented as a vector, relative to the ground truth signal s while accommodating a possible scale mismatch.”
Cracking the cocktail party problem by multi-beam deep attractor network (2017, unknown): “This difference is inline with the output-oracle mismatch in other works such as [45, 46].”