DEEP ATTRACTOR NETWORK FOR SINGLE-MICROPHONE SPEAKER SEPARATION

By Zhuo Chen, Yi Luo, Nima Mesgarani. 2016

426 citations recorded by the source corpus.

Despite the overwhelming success of deep learning in various speech processing tasks, the problem of separating simultaneous speakers in a mixture remains challenging. Two major difficulties in such systems are the arbitrary source permutation and unknown number of sources in the mixture. We propose a novel deep learning framework for single channel speech separation by creating attractor points in high dimensional embedding space of the acoustic signals which pull together the time-frequency bins corresponding to each source. Attractor points in this study are created by finding the centroids of the sources in the embedding space, which are subsequently used to determine the similarity of each bin in the mixture to each source. The network is then trained to minimize the reconstruction error of each source by optimizing the embeddings. The proposed model is different from prior works in that it implements an end-to-end training, and it does not depend on the number of sources in the mixture. Two strategies are explored in the test time, K-means and fixed attractor points, where the latter requires no post-processing and can be implemented in real-time. We evaluated our system on Wall Street Journal dataset and show 5.49% improvement over the previous state-of-the-art methods.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Nima Mesgarani
5. Zhuo Chen
6. Yossi Adi
7. Lior Wolf
8. John R. Hershey

In Signal Processing

1. Shinji Watanabe
2. Jinyu Li
3. Dong Yu
4. John R. Hershey
5. Zhuo Chen
6. Deliang Wang
7. Nima Mesgarani
8. Yossi Adi

Citations in the corpus

• Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “In this work, we focus on fully supervised voice separation using a single microphone, which has seen a great leap in performance following the recent success of deep learning models considering both frequency domain [1, 2, 7, 8, 9, 10], and time-domain [11, 12, 13, 14, 15, 16] modeling.”
• Voice Separation with an Unknown Number of Multiple Speakers (2020, background): “A deep attractor network that extract centroids in the high dimensional embedding spaces, in order to obtain the timefrequency bins for each speaker was presented in (Chen et al., 2017).”
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “ure 4 B show 8 TABLE III COMPARISON WITH OTHER METHODS ON WSJ0-2MIX DATASET. Method # of param. Causal SI-SNRi (dB) SDRi (dB) PESQ DPCL++ [5] 13.6M × 10.8 – – uPIT-BLSTM-ST [6] 92.7M × – 10.0 – DANet [7] 9.1M × 10.5 – 2.64 ADANet [8] 9.1M × 10.4 10.8 2.82 cuPIT-Grid-RD [55] 47.2M × – 10.2 – CBLDNN-G”
• Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “More recent deep learning approaches involve predicting an embedding that encodes the audio clustering [34,35], or optimizing a permutation invariant loss [36].”
• Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “Following DPCL, the DANet approach also learns a high-dimensional embedding of the mixture signals.”
• TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, methodology): “Table 1 shows the performance of our system as well as three state-of-art deep speech separation systems, Deep Clustering method (DPCL) [3], Deep Attractor Network (DANet, [5]) and Permutation Invariant Training (PIT) [4].”
• SDR – Half-baked or Well Done? (2018, methodology): “The scale-invariant SDR (SI-SDR) measure was used in [6, 7, 11, 20–23].”
• End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “The inventions of deep clustering [1, 2, 3], deep attractor networks [4, 5] and permutation free training [1, 2, 6, 7] have dramatically improved the performance of single-channel speaker-independent multispeaker speech separation, demonstrating overwhelming advantages over previous methods includin”
• VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking (2018, methodology): “Secondly, convolutional layers assume time and frequency homogeneity, and thus cannot be applied on an input composed of two completely different signals: a magnitude spectrogram and a speaker embedding.”
• Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network (2018, methodology): “For the separation problem, we evaluated the systems with both SI-SNR improvement (SI-SNRi) and SDR improvement (SDRi) [19] metrics used in [1, 2, 10].”
• Cracking the cocktail party problem by multi-beam deep attractor network (2017, methodology): “orks such as [45, 46]. Finally, when comparing with the single-channel systems, we can see that the proposed system significantly outperforms the single-channel system. And compared with the result in [23, 22, 24], where the test set is generated similarly, the singlechannel system performs much wors”
• CBLDNN-Based Speaker-Independent Speech Separation Via Generative Adversarial Training (2018, background): “Deep attractor is proposed in [9], where network forms attractor points in a high-dimensional embedding space of the signal, and the similarity between attractors and T-F embeddings is converted into a soft separation mask.”
• Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation (2018, methodology): “This dataset has been widely used in many single-channel speech separation studies [1, 2, 26–28] since the de-but of deep clustering.”
• Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization (2021, methodology): “While the number of speakers [43] or maximum number of speakers [40] is fixed for the original DANet, in this paper, we calculated a flexible number of attractors without defining them.”
• FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks (2019, methodology): “2 DANet [3] 10.”
• Listening to Each Speaker One by One with Recurrent Selective Hearing Networks (2018, background): “Unlike DC and DAN, PIT is designed to directly output source separation masks without an explicit clustering step.”
• Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding (2020, background): “perceptual attractors [11], energy-based weighting [12], perceptual weighting filters from speech coding [13], and global masking thresholds [14] [15], while they have not targeted audio coding and model compression.”
• Single Channel Target Speaker Extraction and Recognition with Speaker Beam (2018, background): “Recently, deep clustering [9] and deep attractor networks [11] have been proposed to release these limitations.”
• Continuous Speech Separation: Dataset and Analysis (2020, methodology): “Numerous extentions to these methods were proposed with different focuses [3, 4, 5, 6, 7, 8].”