Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition

By Chris Donahue, Bo Li, Rohit Prabhavalkar. 2017

223 citations recorded by the source corpus.

We investigate the effectiveness of generative adversarial networks (GANs) for speech enhancement, in the context of improving noise robustness of automatic speech recognition (ASR) systems. Prior work [1] demonstrates that GANs can effectively suppress additive noise in raw waveform speech signals, improving perceptual quality metrics; however this technique was not justified in the context of ASR. In this work, we conduct a detailed study to measure the effectiveness of GANs in enhancing speech contaminated by both additive and reverberant noise. Motivated by recent advances in image processing [2], we propose operating GANs on log-Mel filterbank spectra instead of waveforms, which requires less computation and is more robust to reverberant noise. While GAN enhancement improves the performance of a clean-trained ASR system on noisy speech, it falls short of the performance achieved by conventional multi-style training (MTR). By appending the GAN-enhanced features to the noisy inputs and retraining, we achieve a 7% WER improvement relative to the MTR system.

Subfields: Biomedical Engineering, Computational Mechanics, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computational Mechanics

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “Audio enhancement – Deep neural networks have been applied to different audio enhancement tasks, ranging from denoising [17]–[21] to dereverberation [22], [23], lossy coding denoising [2] and frequency bandwidth extension [1], [24].”
SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “Al-ternatively, generative models based on GANs [11, 12] and WaveNet [13] have been proposed.”
Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks (2018, background): “Secondly, many recent generative audio modeling applications, such as text-tospeech [9][10], audio style transfer [12], or speech enhancement [13], output spectrograms (that are typically converted to waveforms using GL), and can potentially benefit from”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “information retrieval, Adversarial Reward Learning (AREL) [163] for visual storytelling, Speech Enhancement GAN (SE-GAN) [164] for speech enhancement, Frequency-Domain Speech Enhancement GAN (FSEGAN) [165] for speech recognition, Caption-GAN [166] for image caption, andImage-to-Poetry GAN (I2P-GAN) ”
Cycle-Consistent Speech Enhancement (2018, methodology): “In speech area, it has been applied to speech enhancement [24, 25, 26, 27], voice conversion [28, 29], acoustic model adaptation [30, 31, 32], noise-robust [33, 34] and speaker-invariant [35, 36] ASR using gradient reversal layer (GRL) [37].”
Deep Learning for Audio Signal Processing (2019, methodology): “In [59], GANs are used to enhance speech represented as logmel spectra.”
Improving GANs for Speech Enhancement (2020, unknown): “There exists a class of generative methods relying on GANs [13], which have been demonstrated to be efficient for speech enhancement [14]–[19].”
Bone-conducted speech enhancement using deep denoising autoencoder (2018, background): “, 2018), as well as those based on the generative adversarial network training criterion (Santiago et al., 2017; Daniel and Tan, 2017; Donahue et al., 2017; Mimura et al., 2017).”
Investigating Generative Adversarial Networks based Speech Dereverberation for Robust Speech Recognition (2018, methodology): “As shown in Table 4, SEGAN degrades the ASR performance within our expectation, which is consistent with the reported results in [20].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “Speech and Audio: GANs have been used for speech and audio analysis such as synthesis [399]–[401], enhancement [402], and recognition [403], .”
Sergan: Speech Enhancement Using Relativistic Generative Adversarial Networks with Gradient Penalty (2019, result): “Similar to the prior works [12,13], we also omitted the latent noise input z as it did not affect the performance.”
Mitigating Noisy Inputs for Question Answering (2019, methodology): “Alternate methods for collecting large scale audio data include Generative Adversarial Networks [11] and manual recording [12].”
Synthesizing Audio with Generative Adversarial Networks (2018, background): “Jansson et al. (2017); Michelsanti & Tan (2017); Donahue et al. (2018) all use GANs in combination with unstructured losses to map spectrograms in one domain to spectrograms in another.”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, methodology): “For in-1 stance, Donahue et al. (2018) applied CGAN on speech enhancement (SE) for automatic speech recognition (ASR); however, the following conclusion was obtained: ”Our experiments indicate that, for ASR, simpler regression approaches may be preferable to GAN-based enhancement.””