Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation

By Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson, Paris Smaragdis. 2015

467 citations recorded by the source corpus.

Monaural source separation is important for many real world applications. It is challenging because, with only a single channel of information available, without any constraints, an infinite number of solutions are possible. In this paper, we explore joint optimization of masking functions and deep recurrent neural networks for monaural source separation tasks, including speech separation, singing voice separation, and speech denoising. The joint optimization of the deep recurrent neural networks with an extra masking layer enforces a reconstruction constraint. Moreover, we explore a discriminative criterion for training neural networks to further enhance the separation performance. We evaluate the proposed system on the TSP, MIR-1K, and TIMIT datasets for speech separation, singing voice separation, and speech denoising tasks, respectively. Our approaches achieve 2.30-4.98 dB SDR gain compared to NMF models in the speech separation task, 2.30-2.48 dB GNSDR gain and 4.32-5.42 dB GSIR gain compared to existing models in the singing voice separation task, and outperform NMF and DNN baselines in the speech denoising task.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Zhuo Chen
5. John R. Hershey
6. Nima Mesgarani
7. Yi Luo
8. Jonathan Le Roux

In Signal Processing

1. Shinji Watanabe
2. Jinyu Li
3. Zhuo Chen
4. John R. Hershey
5. Dong Yu
6. Yi Luo
7. Nima Mesgarani
8. Deliang Wang

Citations in the corpus

• A Wavenet for Speech Denoising (2017, methodology): “A dissimilarity cost was also used for singing voice source separation [11].”
• Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “Finally, we compared with a pretrained separation model [67], which is based on recurrent networks and trained on the TSP dataset [75].”
• TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, background): “In recent years, deep learning approaches have significantly advanced the state of this problem compared to traditional methods [1, 2, 3, 4, 5].”
• Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, background): “Typieally, networks are trained based on parallel sets of mixtures and their constituent target sources [19, 20, 21, 22].”
• Multichannel music separation with deep neural networks (2016, unknown): “Only few of the studies consider the problem of music separation [6], [8], while the others focus on speech separation.”
• Multichannel Audio Source Separation With Deep Neural Networks (2016, methodology): “[10, 11, 12] also used a DBN-DNN and a DRNN to estimate the spectrogram of two sources simultaneously, but they jointly optimized the time-frequency masking function so that the sum of the estimated sources is equal to the mixture.”
• Co-Separating Sounds of Visual Objects (2019, methodology): “Following [21, 18, 50, 52, 31, 13, 8], we start with the commonly adopted “mix-and-separate” idea to selfsupervise source separation.”
• Single Channel Speech Separation with Constrained Utterance Level Permutation Invariant Training Using Grid LSTM (2018, background): “A speaker dependent system in [7] achieves good performance in closed-set speaker-dependent condition.”
• Speaker-Independent Speech Separation With Deep Attractor Network (2017, background): “s, deep learning has made important progress in audio source separation. Specifically, neural networks have been successfully applied in speech enhancement and separation [18], [19], [20], [21], [22], [23], [24] and music separation [25], [26] with significantly better performance than that of traditi”
• Deep Learning for Audio Signal Processing (2019, unknown): “Various deep neural network architectures are applicable in the above settings, including the use of standard methods such as convolutional [121] and recurrent [122] layers.”
• New Era for Robust Speech Recognition (2017, methodology): “It is reported that the factored model in [25] outperforms the unfactored model in [24] in terms of word error rate (WER) on a voice search database with two microphones.”
• Single-Channel Multi-Speaker Separation Using Deep Clustering (2016, unknown): “In contrast to the above, deep learning approaches have recently provided fast and accurate methods on simpler enhancement tasks [17, 18, 19, 20].”