Permutation invariant training of deep models for speaker-independent multi-talker speech separation

By Dong Yu, Morten Kolbæk, Zheng-Hua Tan, Jesper Højvang Jensen. 2016

961 citations recorded by the source corpus.

We propose a novel deep learning training criterion, named permutation invariant training (PIT), for speaker independent multi-talker speech separation, commonly known as the cocktail-party problem. Different from the multi-class regression technique and the deep clustering (DPCL) technique, our novel approach minimizes the separation error directly. This strategy effectively solves the long-lasting label permutation problem, that has prevented progress on deep learning based techniques for speech separation. We evaluated PIT on the WSJ0 and Danish mixed-speech separation tasks and found that it compares favorably to non-negative matrix factorization (NMF), computational auditory scene analysis (CASA), and DPCL and generalizes well over unseen speakers and languages. Since PIT is simple to implement and can be easily integrated and combined with other advanced techniques, we believe improvements built upon PIT can eventually solve the cocktail-party problem.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…(Oord et al., 2016; Luo & Mesgarani, 2019); (2) their training objectives can be either in time-domain (Luo & Mesgarani, 2019), frequency-domain (Yu et al., 2017) or others (Gu et al., 2021; Shen et al., 2023); (3) their designs are inspired by and derived from linguistics and phonetics (Zen et…”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Liu & Wang (2019) suggest to use both PIT and clustering over the frame levels for speaker tracking.”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “Recently, we proposed the Permutation Invariant Training (PIT) technique 1 [38] for attacking the speaker independent multi-talker speech separation problem and showed that PIT effectively solves the label permutation problem.”
Looking to listen at the cocktail party (2018, neutral): “ms using different network configurations. First row (audio-only) is our implementation of a state-of-the-art speech separation model, and shown as a baseline. 1S+Noise 2S clean 2S+Noise 3S clean AO [Yu et al. 2017] 16.0 8.6 10.0 8.6 AV - 1 face 16.0 9.9 10.1 9.1 AV - 2 faces - 10.3 10.6 9.1 AV - 3 ”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “Another permutation-free training scheme was proposed for mask-inference networks first in [1], and was later found to be working very well in [2] and [6].”
Wavesplit: End-to-End Speech Separation by Speaker Clustering (2020, positive): “Contrasting with Permutation Invariant Training, PIT (Yu et al., 2017), this expression does not require searching over the space of permutation since the vectors c = {ci}i=1 is consistent with the order of the labels {y}i=1 as explained above.”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “…includes monau-ral speech separation methods, which address the problem of separating a mixture of speakers from a single audio track [23, 36, 60, 67], as well as monaural speech enhancement methods, which tackle the problem of removing non-speech background sounds [8, 30, 42, 55, 56, 63] and…”
Improving Sound Event Detection in Domestic Environments using Sound Separation (2020, methodology): “Some tasks are trained with permutationinvariant training (PIT) [28] or groupwise PIT.”
FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing (2019, background): “vocoder Y 1c S 1c Utterance-level permutation invariant training (uPIT) is always applied to address the output permutation problem [36,37]. source C , iteration i source C , iteration i where Y c , Y ∗ c ∈ R T × F are the magnitude spectrograms of the target and estimated signals respectively, and…”
Complex Neural Spatial Filter: Enhancing Multi-Channel Target Speech Separation in Complex Domain (2021, methodology): “In the early study, the learning objective was to estimate a magnitude mask of the target speech from the mixture magnitude, leading to ratio mask (RM) based TSS methods [9], [10].”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, background): “The loss function of chimera++ networks is a weighted combination of the above two loss functions: (8) At run time, the PIT output is utilized for separation.”
Single Channel Target Speaker Extraction and Recognition with Speaker Beam (2018, background): “Permutation invariant training [12] is another approach, which mitigates the frame level permutation problem at the training stage, by modifying the training objective function such that labels are permuted to find the closest match with the output of the DNN.”
Continuous Speech Separation with Conformer (2020, positive): “Started from Deep Clustering (DC) [11, 12] and Permutation Invariant Training (PIT) [13, 14], a series of separation models have been shown effective in handling overlapped speech.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, neutral): “For example, the dataset presented in [239] has been collected using four different dummy heads and used for SSL in [103].”
A Survey on Deep Learning (2018, methodology): “In [174], the authors implement the speech separation model with the permutation-invariant training technique.”