End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks

By Szu-Wei Fu, Tao-Wei Wang, Yu Tsao, Xugang Lu, Hisashi Kawai. 2017

297 citations recorded by the source corpus.

Speech enhancement model is used to map a noisy speech to a clean speech. In the training stage, an objective function is often adopted to optimize the model parameters. However, in the existing literature, there is an inconsistency between the model optimization criterion and the evaluation criterion for the enhanced speech. For example, in measuring speech intelligibility, most of the evaluation metric is based on a short-time objective intelligibility (STOI) measure, while the frame based mean square error (MSE) between estimated and clean speech is widely used in optimizing the model. Due to the inconsistency, there is no guarantee that the trained model can provide optimal performance in applications. In this study, we propose an end-to-end utterance-based speech enhancement framework using fully convolutional neural networks (FCN) to reduce the gap between the model optimization and the evaluation criterion. Because of the utterance-based optimization, temporal correlation information of long speech segments, or even at the entire utterance level, can be considered to directly optimize perception-based objective functions. As an example, we implemented the proposed FCN enhancement framework to optimize the STOI measure. Experimental results show that the STOI of a test speech processed by the proposed approach is better than conventional MSE-optimized speech due to the consistency between the training and the evaluation targets. Moreover, by integrating the STOI into model optimization, the intelligibility of human subjects and automatic speech recognition system on the enhanced speech is also substantially improved compared to those generated based on the minimum MSE criterion.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “tly designed to replace STFT and iSTFT. Alternatively, feature extraction can be implicitly incorporated into the network architecture, for example by using a convolutional neural network layer (CNN) [39], [40]. These methods are different in how they extract features from the waveform and in terms ”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, background): “D CNNs, have difﬁculty on learning such long-term temporal dependency [12].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Another recent trend is to avoid the phase inconsistency problem altogether by operating in the time domain, using convolutional neural networks [30, 31], WaveNet [32], generative adversarial networks [33], or encoder-decoder architectures [34].”
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (2020, background): “The time-domain approaches can further fall into two categories — direct regression [2,3] and adaptive front-end approaches [4–6].”
Convolutional Neural Networks to Enhance Coded Speech (2018, methodology): “, log-power spectrum [32]–[34] or complex spectrogram [35], or a mapping directly between the noisy raw speech waveform and clean raw speech waveform [36], [37].”
MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement (2021, methodology): “The design of these two metrics considers human auditory perception and has shown higher correlation to subjective listening tests than simple 𝐿 1 or 𝐿 2 distance between clean and degraded speech signals [1, 4].”
A New Framework for CNN-Based Speech Enhancement in the Time Domain (2019, background): “Similar to our work, a time domain network has been proposed using a loss based on a short-term objective intelligibility metric [14].”
Conditional Diffusion Probabilistic Model for Speech Enhancement (2022, methodology): “The ﬁrst category trains discriminative models to minimize the difference be-tween enhanced and clean speech, where the difference can be a point-wise L p -norm distance [8], or can be computed based on a perceptual metric [9, 10].”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “A discriminative method minimizes the difference between enhanced and clean speech [13, 16, 17, 42], while generative models are optimized by estimating the distribution of clean signals [51, 80, 86, 106, 108].”
Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement (2019, result): “We also compare our proposed approach with two recent time-domain speech enhancement approaches: AECNN-SM (autoencoder CNN with STFT magnitude loss) [24] and FCN (fully convolutional network) [8].”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, methodology): “In this study, to solve the drawbacks of the abovementioned methods and the DEM problem, the discriminator in GAN is associated with the evaluation metrics of interest (Although these evaluation functions are complex, Fu et al. (2018a) showed that they can be approximated by neural networks).”
Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding (2020, background): “Other neural speech enhancement systems implement short-time objective intelligibility (STOI) [16] and perceptual evaluation of speech quality (PESQ) [17] as the loss [18], [19].”
Speech Enhancement with a Generative Adversarial Network (2019, background): “Fu et al. (2018) pointed out that there has been a mismatch between the training’s loss function and the evaluation criterion used on the enhanced speech.”
Bone-conducted speech enhancement using deep denoising autoencoder (2018, background): “, 2017), and fully convolutional networks (Fu et al., 2018).”