Real Time Speech Enhancement in the Waveform Domain

By Alexandre D'efossez, Gabriel Synnaeve, Yossi Adi. 2020

636 citations recorded by the source corpus.

We present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is optimized on both time and frequency domains, using multiple loss functions. Empirical evidence shows that it is capable of removing various kinds of background noise including stationary and non-stationary noises, as well as room reverb. Additionally, we suggest a set of data augmentation techniques applied directly on the raw waveform which further improve model performance and its generalization abilities. We perform evaluations on several standard benchmarks, both using objective metrics and human judgements. The proposed model matches state-of-the-art performance of both causal and non causal methods while working directly on the raw waveform.

Subfields: Artificial Intelligence, Computational Mechanics, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Speech & Audio

In Signal Processing

Citations in the corpus

High Fidelity Neural Audio Compression (2022, background): “…framework was shown to provide great results in various audio-related tasks, e.g., source separation and enhancement (Défossez et al., 2019; Defossez et al., 2020), neural vocoders (Kumar et al., 2019; Kong et al., 2020), audio codec (Zeghidour et al., 2021), and artiﬁcial bandwidth…”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “We also consider Demucs [11], a SOTA speech enhancement model trained with regression and adversarial losses for denoising experiments.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, background): “The baseline is enhanced to account for recent improvement from speech source separation (Défossez et al., 2020) and pitch/tempo shift augmentation (Cohen-Hadria et al.”
Music Source Separation in the Waveform Domain (2019, background): “Note that the same positive impact of resampling has been noted for speech denoising [Defossez et al., 2020].”
Dnsmos: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2020, background): “Recently, researchers have seen success in learning features within the model for tasks such as SE [17], speech, and music synthesis [18] and to learn acoustic models [19].”
Conditional Diffusion Probabilistic Model for Speech Enhancement (2022, methodology): “We were able to show that CDiffuSE maintains strong performance when regression-based approaches such as Demucs and Conv-TasNet collapse.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “In Table 7 we compare the best enhancement systems available in SpeechBrain against the SotA DEMUCS model [95] on the Voicebank-DEMAND corpus [96].”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, result): “For comparison with HiFi-GAN (Kong et al., 2020a), MelGAN (Kumar et al., 2019), Waveglow (Prenger et al., 2019), TFilm (Birnbaum et al., 2019), VoiceFixer (Liu et al., 2021), MetricGAN+ (Fu et al., 2021), DEMUCS (D´efossez et al., 2019) we employ the ofﬁcial implementations provided by the authors.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “We applied a denoiser6 (Defossez et al., 2020) to the speech of VoxPopuli and Common Voice as the speech preprocessing to increase signal-to-noise ratio (SNR) given that they are noisier than CSS10 audios.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, neutral): “A discriminative method minimizes the difference between enhanced and clean speech [13, 16, 17, 42], while generative models are optimized by estimating the distribution of clean signals [51, 80, 86, 106, 108].”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “We use a denoiser model [Defossez et al., 2020] to remove background music.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, methodology): “We used the speech enhancement model proposed by (Defossez et al., 2020) where the ith convolutional layer has 2i−1 ∗ 64 output channels.”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, result): “Accordingly, we decide not to compare with [38] but compare with a publicly available strong generation-based audio enhancement model, Demucs [12], which is trained on ∼ 650 hours of high quality clean speech mixed with noise from AudioSet [20] and FreeSound 4 .”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, methodology): “This includes monaural speech separation methods, which address the problem of separating a mixture of speakers from a single audio track [23, 36, 60, 66], as well as monaural speech enhancement methods, which tackle the problem of removing nonspeech background sounds [8, 30, 42, 55, 56, 63] and rev”
High Fidelity Speech Regeneration with Application to Speech Enhancement (2021, methodology): “Due to the success of deep networks, there has been a growing interest in deep learningbased methods for both speech enhancement and dereverberation working on either frequency domain or directly on the raw waveform [3, 4, 5].”
Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “For L, we use L because several studies showed that this loss function can stabilize the adversarial training [35, 36, 54].”
Dnsmos P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2021, background): “Recently, researchers have seen success in learning features within the model for tasks such as SE [25], speech, and music synthesis [26] and to learn acoustic models [27].”
Universal Speech Enhancement with Score-based Diffusion (2022, methodology): “…3.42 FullSubNet (Hao et al., 2021) 3.31 2.78 0.96 DCCRN+ (Lv et al., 2021) 2.84 3.33 CTS-Net (Li et al., 2021a) 2.92 3.59 3.42 2.94 0.97 Demucs (Défossez et al., 2020) 3.07 3.63 0.95 SN-Net (Zheng et al., 2021) 3.12 3.60 3.39 SE-Conformer (Kim & Seo, 2021) 3.12 3.82 0.95 MetricGAN+ (Fu et al.,…”