Speech Enhancement and Dereverberation With Diffusion-Based Generative Models

By Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, Timo Gerkmann. 2022

389 citations recorded by the source corpus.

In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that is based on a stochastic differential equation and delve into an extensive theoretical examination of its implications. Opposed to usual conditional generation tasks, we do not start the reverse process from pure Gaussian noise but from a mixture of noisy speech and Gaussian noise. This matches our forward process which moves from clean speech to noisy speech by including a drift term. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. By adapting the network architecture, we are able to significantly improve the speech enhancement performance, indicating that the network, rather than the formalism, was the main limitation of our original approach. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models and achieves better generalization when evaluating on a different corpus than used for training. We complement the results with an instrumental evaluation using real-world noisy recordings and a listening experiment, in which our proposed method is rated best. Examining different sampler configurations for solving the reverse process allows us to balance the performance and computational speed of the proposed method. Moreover, we show that the proposed method is also suitable for dereverberation and thus not limited to additive background noise removal.

Subfields: Computational Mechanics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Matt Le
2. Fredrik K. Gustafsson
3. Thomas Bo Schön
4. In-So Kweon
5. Wei-Ning Hsu

In Signal Processing

1. Bowen Shi
2. Wei-Ning Hsu
3. Apoorv Vyas
4. Matt Le
5. Andros Tjandra
6. Fredrik K. Gustafsson
7. Thomas Bo Schön
8. Alexander H. Liu

Citations in the corpus

• Generative Pre-training for Speech with Flow Matching (2023, methodology): “On the other hand, examples for distribution mapping include diffusion models that transform noisy speech to clean speech for denoising (Lu et al., 2021; 2022; Richter et al., 2023), or speech mixture to non-overlapping speech for separation (Scheibler et al., 2023).”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, positive): “SGMSE+ [89] further extends SGMSE [124] with detailed theoretical analysis and improved model architecture.”
• Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, methodology): “Most similar to the IR-SDE is the work of Welker et al. (2022b) and Richter et al. (2022), in which a mean-reverting SDE is applied to the speech processing tasks of speech enhancement and speech dereverberation.”