On Training Targets for Supervised Speech Separation

By Yuxuan Wang, Arun Narayanan, Deliang Wang. 2014

1,082 citations recorded by the source corpus.

Formulation of speech separation as a supervised learning problem has shown considerable promise. In its simplest form, a supervised learning algorithm, typically a deep neural network, is trained to learn a mapping from noisy features to a time-frequency representation of the target of interest. Traditionally, the ideal binary mask (IBM) is used as the target because of its simplicity and large speech intelligibility gains. The supervised learning framework, however, is not restricted to the use of binary targets. In this study, we evaluate and compare separation results by using different training targets, including the IBM, the target binary mask, the ideal ratio mask (IRM), the short-time Fourier transform spectral magnitude and its corresponding mask (FFT-MASK), and the Gammatone frequency power spectrum. Our results in various test conditions reveal that the two ratio mask targets, the IRM and the FFT-MASK, outperform the other targets in terms of objective intelligibility and quality metrics. In addition, we find that masking based targets, in general, are significantly better than spectral envelope based targets. We also present comparisons with recent methods in non-negative matrix factorization and speech enhancement, which show clear performance advantages of supervised speech separation.

Subfields: Computational Mechanics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Signal Processing

Citations in the corpus

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “Following [69] and [70], instead of directly predicting the source STFTs, we firstly estimate a group of masks {Ms}s=1 with a deep learning model, and then obtain each source STFT with Xs = Ms Y, where is an elementwise product.”
Time-Frequency Masking in the Complex Domain for Speech Dereverberation and Denoising (2017, positive): “A variant of this feature set has been shown to be effective for speech separation [38], and they have recently been shown to work well for cIRM estimation [42].”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, background): “revious LSTM-TasNet. Moreover, the separation accuracy of Conv-TasNet surpasses the performance of ideal time-frequency masks, including the ideal binary mask (IBM [28]), ideal ratio mask (IRM, [29], [30]), and Winener ﬁlterlike mask (WFM, [31]). The rest of the paper is organized as follows. We bri”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “The Ideal Ratio Mask (IRM) [27] for each source is deﬁned as When the phase of Y is used for reconstruction, the IRM achieves the highest Signal to Distortion Ratio (SDR) [44], when all sources have the same phase, (which is an in-valid assumption in general).”
Looking to listen at the cocktail party (2018, neutral): “In previous work [Wang and Chen 2017; Wang et al. 2014], multiplicative masks have been observed to work better than alternatives such as direct prediction The visual streams take as input thumbnails of detected faces in each frame in the video, and the audio stream takes as input the video’s…”
Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, methodology): “Typieally, networks are trained based on parallel sets of mixtures and their constituent target sources [19, 20, 21, 22].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Sigmoidal units are dominantly used in the output layer of deep learning based T-F masking [36, 35], partly because they can model well data with bi-modal distribution [37], such as the IRM [38] and its variants [36].”
Speech Enhancement in Multiple-Noise Conditions Using Deep Neural Networks (2016, background): “Mel-frequency spectrum [14] [16], ideal binary mask, ideal ratio mask, short-time Fourier transform magnitude and its mask [21] [22], log-power spectra are all potential candidates.”
Speech Dereverberation Using Fully Convolutional Networks (2018, methodology): “Following [18] we use the log-spectrum as an effective feature vector.”
Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement (2019, background): “For supervised speech enhancement, a proper selection of the training target is important [38].”
Ultra Fast Speech Separation Model with Teacher Student Learning (2021, background): “Following [11, 12], instead of directly outputting the STFT of the individual signals [X1(t, f) .”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, background): “They typically estimate a real or complex ratio mask for each T-F bin to map noisy spectra to speech spectra (Williamson, Wang, and Wang 2015; Wang, Narayanan, and Wang 2014; Choi et al. 2019) or directly predict the speech representation (Xu et al. 2013; Tan and Wang 2018).”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, background): “Notable works include masking based speech enhancement studies [1]–[3], and speaker separation studies such as deep clustering (DC) [4]–[6] and permutation invariant training (PIT) [7], [8].”
Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement (2017, methodology): “…a multi-objective learning framework is proposed to optimize a joint objective function, encompassing errors not only for the primary clean LPS features but also errors in secondary targets for continuous features, such as MFCC, and for categorical information, such as ideal binary mask (IBM) [22].”
Single-Channel Multi-Speaker Separation Using Deep Clustering (2016, result): “In contrast to the above, deep learning approaches have recently provided fast and accurate methods on simpler enhancement tasks [17, 18, 19, 20].”
Improved MVDR Beamforming Using Single-Channel Mask Prediction Networks (2016, background): “It has been shown in earlier studies of source separation that it is beneficial for estimating the target signal to predict a mask that multiplies the STFT of the mixed signal [12, 20, 9].”
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (2020, background): “In the masking family, ideal binary mask (IBM) [20], ideal ratio mask (IRM) [10] and spectral magnitude mask (SMM) [21] only use the magnitude between clean speech and mixture speech, ignoring the phase information.”