Universal Speech Enhancement with Score-based Diffusion

By Joan Serrà, Santiago Pascual, Jordi Pons, Recep Oguz Araz, Davide Scaini. 2022

146 citations recorded by the source corpus.

Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings. Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous. In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time. Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks. We show that this approach significantly outperforms the state of the art in a subjective test performed by expert listeners. We also show that it achieves competitive objective scores with just 4-8 diffusion steps, despite not considering any particular strategy for fast sampling. We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.

Subfields: Biomedical Engineering, Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Takuya Yoshioka
2. Wen Wang

In Cognitive Neuroscience

1. Zhuo Chen
2. Jinyu Li
3. Bowen Shi
4. Soujanya Poria
5. Jacob Donley
6. Tal Remez
7. Yossi Adi
8. Wei-Ning Hsu

In Signal Processing

1. Jinyu Li
2. Zhuo Chen
3. Shujie Liu
4. Yossi Adi
5. Sanyuan Chen
6. Takuya Yoshioka
7. Wei-Ning Hsu
8. Naoyuki Kanda

Citations in the corpus

• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “However, these prior methods typically required distinct expert models for each task, which is not ideal, given the potential diversity of acoustic disturbances [26].”
• Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, background): “Diffusion models have emerged as a promising approach for speech processing, particularly in speech enhancement [21, 37, 29, 22].”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “results show that this approach achieves the state-of-the-art performance according to a subjective text with expert listeners [96].”
• NADiffuSE: Noise-aware Diffusion-based Model for Speech Enhancement (2023, methodology): “And then conditioned on the coarsely enhanced signal, the diffusion model is used to generate the fine-grained enhanced signal [28], [29] or the residual signal [27] against the clean one.”
• ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “Second, regardless of the training objective (regression [40], adversarial [43, 45, 50], or score-matching [48]), previous universal models predict reference clean speech directly.”
• Speech Enhancement and Dereverberation With Diffusion-Based Generative Models (2022, unknown): “…i.e. a diffusion-based vocoder network is used to synthesize clean speech by sampling from an unconditional prior, while a conditioner network takes noisy speech as input and performs the core part of denoising by providing enhanced speech representations to the vocoder network [23], [30].”