Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement

By Yong Xu, Jun Du, Zhen Huang, Lirong Dai, Chin-Hui Lee. 2017

111 citations recorded by the source corpus.

We propose a multi-objective framework to learn both secondary targets not directly related to the intended task of speech enhancement (SE) and the primary target of the clean log-power spectra (LPS) features to be used directly for constructing the enhanced speech signals. In deep neural network (DNN) based SE we introduce an auxiliary structure to learn secondary continuous features, such as mel-frequency cepstral coefficients (MFCCs), and categorical information, such as the ideal binary mask (IBM), and integrate it into the original DNN architecture for joint optimization of all the parameters. This joint estimation scheme imposes additional constraints not available in the direct prediction of LPS, and potentially improves the learning of the primary target. Furthermore, the learned secondary information as a byproduct can be used for other purposes, e.g., the IBM-based post-processing in this work. A series of experiments show that joint LPS and MFCC learning improves the SE performance, and IBM-based post-processing further enhances listening quality of the reconstructed speech.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

Real Time Speech Enhancement in the Waveform Domain (2020, background): “Authors in [43] suggested to use a multi-objective loss function to further improve speech quality.”
Deep learning for minimum mean-square error approaches to speech enhancement (2019, methodology): “The results from Section 6.3 indicate that most of the erformance improvement gained by the MMSE-LSA estimator utilising he ResBLSTM a priori SNR estimator over Xu2017 is due to the training arget, and not the model, training strategy, or post-processing.”
SNR-Aware Convolutional Neural Network Modeling for Speech Enhancement (2016, background): “Xu et al. [21] proposed a DNN structure by estimating clean LPS and MFCC features together to achieve better enhancement performance.”
NADiffuSE: Noise-aware Diffusion-based Model for Speech Enhancement (2023, background): “In the last decade, deep learning (DL) methods [1], [2] have become the mainstream for speech enhancement (SE), which can be divided into two categories: the discriminative and generative ones [3].”
Rapid adaptation for deep neural networks through multi-task learning (2015, background): “MLT has been proposed for improving the generalization capability of classiﬁers, and it has been adopted in various speech re-lated tasks, such as isolated digit recognition [23, 24], phoneme recognition [25], speech synthesis [26] and speech enhancement [27].”
DeepMMSE: A Deep Learning Approach to MMSE-Based Noise Power Spectral Density Estimation (2020, background): “…that estimates the ideal ratio mask (IRM) [57], and Xu2017 is a multi-layer perceptron (MLP) clean speech spectrum estimator that incorporates multi-objective learning and ideal binary mask (IBM) post-processing (available from https://github.com/ yongxuUSTC/DNN-for-speech-enhancement) [58].”
Deep Residual-Dense Lattice Network for Speech Enhancement (2020, methodology): “Multi-layer perceptrons (MLPs) were amongst the first artificial neural networks (ANNs) used for speech enhancement (Xu et al. 2017).”