SEGAN: Speech Enhancement Generative Adversarial Network

By Santiago Pascual, Antonio Bonafonte, Joan Serrà. 2017

1,295 citations recorded by the source corpus.

Current speech enhancement techniques operate on the spectral domain and/or exploit some higher-level feature. The majority of them tackle a limited number of noise conditions and rely on first-order statistics. To circumvent these issues, deep networks are being increasingly used, thanks to their ability to learn complex functions from large example sets. In this work, we propose the use of generative adversarial networks for speech enhancement. In contrast to current techniques, we operate at the waveform level, training the model end-to-end, and incorporate 28 speakers and 40 different noise conditions into the same model, such that model parameters are shared across them. We evaluate the proposed model using an independent, unseen test set with two speakers and 20 alternative noise conditions. The enhanced samples confirm the viability of the proposed model, and both objective and subjective evaluations confirm the effectiveness of it. With that, we open the exploration of generative architectures for speech enhancement, which may progressively incorporate further speech-centric design choices to improve their performance.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Synthesizing Tabular Data using Generative Adversarial Networks (2018, background): “Other GAN applications include information retrieval [45], dialogue systems [27], and speech processing [32].”
Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “In the field of audio denoising, better qualitative results are obtained computing the reconstruction loss in some feature space [9], and using adversarial losses [10], hence a similar approach has also been adopted for audio superresolution [11, 12].”
SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “Al-ternatively, generative models based on GANs [11, 12] and WaveNet [13] have been proposed.”
Audio Codec Enhancement with Generative Adversarial Networks (2020, methodology): “GANs have achieved tremendous success in the field of computer vision to generate high-resolution diverse images from complex datasets [18], image restoration [19], as well as for speech enhancement [20].”
Speech Denoising with Deep Feature Losses (2018, methodology): “, L(1) loss on the raw waveform); pipelines with more advanced loss functions have shown limited gains in mismatched conditions [23, 24].”
Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (2017, methodology): “[1] demonstrate promising performance of GANs for speech enhancement in the presence of additive noise, posing enhancement as a translation task from noisy signals to clean ones.”
A Wavenet for Speech Denoising (2017, methodology): “The dataset we used [12, 18] was generated from two sources: speech data was supplied by the Voice Bank corpus [19] while environmental sounds were provided by the Diverse Environments Multichannel Acoustic Noise Database (DEMAND) [20].”
Real Time Speech Enhancement in the Waveform Domain (2020, methodology): “Over the last years, there has been a growing interest towards DNN based methods for speech enhancement [34, 35, 36, 37, 20, 38, 39, 22, 40, 7, 41, 8, 42, 21, 37].”
Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, methodology): “…to the previous work [11], we deﬁne a single STFT loss as follows: 3 Our method adopts least-squares GANs thanks to its training stability [19–22]. where ˆ x denotes the generated sample (i.e., G ( z ) ), and L sc and L mag denote spectral convergence and log STFT magnitude loss,…”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, background): “The choice of L1 is driven by robustness, as the speech distribution is very peaky and zero-centered with prominent outliers [27].”
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, neutral): “2 This framework adopts a least-squares GANs thanks to its stability during the training process [22, 24, 27, 28].”
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, methodology): “In recent years, this approach has been used successfully applied in tasks such as speech recognition, speech synthesis and speech enhancement [9, 10, 11, 12, 13], but waveform-level speech separation with deep learning has not been investigated yet.”
Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation (2018, methodology): “• We replace strided transposed convolution used in previous work [7, 16] for upsampling feature maps with linear interpolation followed by a normal convolution to avoid artifacts.”
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models (2017, background): “Empirically, GANs generate excellent samples of natural images (Radford, Metz, and Chintala 2015), audio signals (Pascual, Bonafonte, and Serrà 2017), and of behaviors in imitation learning (Ho and Ermon 2016; Li, Song, and Ermon 2017).”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Another recent trend is to avoid the phase inconsistency problem altogether by operating in the time domain, using convolutional neural networks [30, 31], WaveNet [32], generative adversarial networks [33], or encoder-decoder architectures [34].”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “This includes monaural speech separation methods, which address the problem of separating a mixture of speakers from a single audio track [23, 36, 60, 66], as well as monaural speech enhancement methods, which tackle the problem of removing nonspeech background sounds [8, 30, 42, 55, 56, 63] and rev”
Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images (2018, methodology): “The architecture of the Dynamic GM-GAN is modiﬁed so that G receives as input a categorical random variable k , which determines from which Gaussian the latent vector should be sampled.”