Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification

By Daniel Michelsanti, Zheng-Hua Tan. 2017

224 citations recorded by the source corpus.

Improving speech system performance in noisy environments remains a challenging task, and speech enhancement (SE) is one of the effective techniques to solve the problem. Motivated by the promising results of generative adversarial networks (GANs) in a variety of image processing tasks, we explore the potential of conditional GANs (cGANs) for SE, and in particular, we make use of the image processing framework proposed by Isola et al. [1] to learn a mapping from the spectrogram of noisy speech to an enhanced counterpart. The SE cGAN consists of two networks, trained in an adversarial manner: a generator that tries to enhance the input noisy spectrogram, and a discriminator that tries to distinguish between enhanced spectrograms provided by the generator and clean ones from the database using the noisy spectrogram as a condition. We evaluate the performance of the cGAN method in terms of perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and equal error rate (EER) of speaker verification (an example application). Experimental results show that the cGAN method overall outperforms the classical short-time spectral amplitude minimum mean square error (STSA-MMSE) SE algorithm, and is comparable to a deep neural network-based SE approach (DNN-SE).

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (2017, methodology): “[20] employ GANs for SFM, but target speaker verification rather than ASR.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “We augment our audio-visual network with a u-net encoder-decoder [43,69,70] that maps the mixture sound to its on- and off-screen components (Figure 6).”
rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method (2019, background): “Recently DNN based speech enhancement methods have also been proposed for improving speech intelligibility [55], automatic speech recognition [56] and speaker verification [57, 58].”
Speech Dereverberation Using Fully Convolutional Networks (2018, methodology): “The objective is the same as in [19] and [21]:”
Investigating Generative Adversarial Networks based Speech Dereverberation for Robust Speech Recognition (2018, methodology): “Moreover, we discover that, using reverberant spectrogram as a condition toD, as suggested in [20, 21], may degrade the performance ofG.”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, methodology): “Alternatively, some AV models, such as [195], have been trained with the mean absolute error (MAE), experimentally proved to increase the spectral detail of the estimates and obtain higher performance if compared to MSE [175], [198].”
Supervised Speech Separation Based on Deep Learning: An Overview (2017, unknown): “In another GAN study [122], G tries to enhance the spectrogram of noisy speech while D tries to distinguish between the enhanced spectrograms and those of clean speech.”
Speech Enhancement with a Generative Adversarial Network (2019, background): “8 Conditional Generative Adversarial Networks The conditional version of generative adversarial nets was introduced by Mirza and Osindero (2014). In an unconditioned generative model, the goal is to generate samples from the training data’s distribution pdata.”
Deep Learning for Environmentally Robust Speech Recognition (2017, unknown): “The adversarial training strategy was examined in [81] and [82], and was found to perform superior to other traditional approaches, such as Wiener filtering.”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, background): “…applications of image (Ledig et al., 2017; Wang et al., 2018) and speech processing (Pandey & Wang, 2018; Wang & Chen, 2018; Donahue et al., 2018; Michelsanti & Tan, 2017) show that this loss term provides very marginal improvement (sometimes even degrade the performance) in terms of objective…”