Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks

By Morten Kolbaek, Dong Yu, Zheng-Hua Tan, Jesper Højvang Jensen. 2017

782 citations recorded by the source corpus.

In this paper, we propose the utterance-level permutation invariant training (uPIT) technique. uPIT is a practically applicable, end-to-end, deep-learning-based solution for speaker independent multitalker speech separation. Specifically, uPIT extends the recently proposed permutation invariant training (PIT) technique with an utterance-level cost function, hence eliminating the need for solving an additional permutation problem during inference, which is otherwise required by frame-level PIT. We achieve this using recurrent neural networks (RNNs) that, during training, minimize the utterance-level separation error, hence forcing separated frames belonging to the same speaker to be aligned to the same output stream. In practice, this allows RNNs, trained with uPIT, to separate multitalker mixed speech without any prior knowledge of signal duration, number of speakers, speaker identity, or gender. We evaluated uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks and found that uPIT outperforms techniques based on nonnegative matrix factorization and computational auditory scene analysis, and compares favorably with deep clustering, and the deep attractor network. Furthermore, we found that models trained with uPIT generalize well to unseen speakers and languages. Finally, we found that a single model, trained with uPIT, can handle both two-speaker, and three-speaker speech mixtures.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shinji Watanabe
2. Jinyu Li
3. Dong Yu
4. Zhuo Chen
5. Yossi Adi
6. Lior Wolf
7. Nima Mesgarani
8. Yoshua Bengio

In Speech & Audio

1. Shinji Watanabe
2. Jinyu Li
3. Deliang Wang
4. John R. Hershey
5. Yossi Adi
6. Zhuo Chen
7. Nima Mesgarani
8. Yi Luo

In Signal Processing

1. Shinji Watanabe
2. Jinyu Li
3. Zhuo Chen
4. Deliang Wang
5. Dong Yu
6. John R. Hershey
7. Yossi Adi
8. Nima Mesgarani

Citations in the corpus

• SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “ed on GANs [11, 12] and WaveNet [13] have been proposed. Speech denoising can also be seen as a special case of source separation, in which one of the sources represents the speech signal of interest [14, 15, 16, 17]. Our work belongs to the family of multi-modal models, which leverage additional co”
• Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “To tackle the permutation invariant problem we use the utterance level Permutation Invariant Training (uPIT) loss, as proposed in [7]:”
• Voice Separation with an Unknown Number of Multiple Speakers (2020, background): “The loss `(s, ŝ) is often denoted as the utterance level permutation invariant training (uPIT) (Kolbæk et al., 2017).”
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, neutral): “M – – 13.0 13.5 3.33 WFM – – 13.4 13.8 3.70 TABLE IV COMPARISON WITH OTHER SYSTEMS ON WSJ0-3MIX DATASET. Method # of param. Causal SI-SNRi (dB) SDRi (dB) PESQ DPCL++ [5] 13.6M × 7.1 – – uPIT-BLSTM-ST [6] 92.7M × – 7.7 – DANet [7] 9.1M × 8.6 8.9 1.92 ADANet [8] 9.1M × 9.1 9.4 2.16 Conv-TasNet-gLN 8.9”
• TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, methodology): “Under the noncausal configuration, our system outperforms the two-stage system with a separation network and a mask enhancement network proposed in [4], but is slightly worse than DPCL.”
• SDR – Half-baked or Well Done? (2018, methodology): “Both SI-SDR and BSS eval ’s SDR have recently been used by various studies [6–9,11,21–23,25,26] in the context of single-channel speaker-independent speech separation on the wsj0-2mix dataset [6], some of these studies reporting both figures [21–23 ,25].”
• Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, methodology): “All models are trained with utterance-level permutation invariant training (uPIT) [29] to maximize scale-invariant SNR (SI-SNR) [8].”
• Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR (2020, methodology): “Most separation and multi-talker ASR approaches assume that the number of talkers is known [3, 4, 6, 10, 13–15], although this is not the case in realistic situations.”
• Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “It should be noted in LPIT, we apply permutation invariant training (PIT) [19, 20] to remedy the source permutation problem, while the permutation in T-S LLTS loss is determined by the teacher model.”
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “Although some AV-SS systems used PIT or DC (cf. Table IV), source permutation is less of a problem in AVSS, assuming that the target speakers are visible while they talk: visual information is a strong guidance for the systems and allows to automatically assign the separated speech signals to the co”
• Speaker-Independent Speech Separation With Deep Attractor Network (2017, methodology): “4 uPIT-BLSTM-ST [34] 2 7.”
• Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network (2018, methodology): “For the separation problem, we evaluated the systems with both SI-SNR improvement (SI-SNRi) and SDR improvement (SDRi) [19] metrics used in [1, 2, 10].”
• Recent Developments on Espnet Toolkit Boosted By Conformer (2020, methodology): “Both models are trained with uPIT [20] based on Phase Sensitive Masks (PSM) and ReLU activation function.”
• End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation (2019, methodology): “All models are trained to maximize scale-invariant SNR (SI-SNR) [25] with utterance-level permutation invariant training (uPIT) [26].”
• Single Channel Speech Separation with Constrained Utterance Level Permutation Invariant Training Using Grid LSTM (2018, methodology): “Recently, permutation invariant training (PIT) [11] and utterance level PIT (uPIT) [12] were introduced to solve the speech separation problem with end-to-end training.”
• FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks (2019, methodology): “In order to solve tracing and permutation problem, the PIT training criteria [12, 35] is employed in this work.”
• The Phasebook: Building Complex Masks via Discrete Representations for Source Separation (2019, background): “The field of speech separation and speech enhancement has witnessed dramatic improvements in performance with the recent advent of deep learning-based techniques [1–11].”
• Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, methodology): “Permutation-invariant training (PIT) is utilized to determine an appropriate permutation in terms of the order of sources [27].”