Phase-aware Speech Enhancement with Deep Complex U-Net

By Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, Kyogu Lee. International Conference on Learning Representations, 2019

401 citations recorded by the source corpus.

Most deep learning-based models for speech enhancement have mainly focused on estimating the magnitude of spectrogram while reusing the phase from noisy speech for reconstruction. This is due to the difficulty of estimating the phase of clean speech. To improve speech enhancement performance, we tackle the phase estimation problem in three ways. First, we propose Deep Complex U-Net, an advanced U-Net structured model incorporating well-defined complex-valued building blocks to deal with complex-valued spectrograms. Second, we propose a polar coordinate-wise complex-valued masking method to reflect the distribution of complex ideal ratio masks. Third, we define a novel loss function, weighted source-to-distortion ratio (wSDR) loss, which is designed to directly correlate with a quantitative evaluation measure. Our model was evaluated on a mixture of the Voice Bank corpus and DEMAND database, which has been widely used by many deep learning models for speech enhancement. Ablation experiments were conducted on the mixed dataset showing that all three proposed approaches are empirically valid. Experimental results show that the proposed method achieves state-of-the-art performance in all metrics, outperforming previous approaches by a large margin.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Signal Processing

View Paper: https://www.semanticscholar.org/paper/101757dd6c7cb3704a54b25fd8ae0e8c0daee248

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Signal Processing

Citations in the corpus

Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training (2020, methodology): “As for the de-noising baseline with external speech enhancement module, we use the state-of-the-art speech enhancement model named DCUnet [26] to de-noise F-N and M-N.”
Speech Enhancement and Dereverberation With Diffusion-Based Generative Models (2022, neutral): “In addition to the main feature extraction path of the multi-resolution U-Net structure, the network incorporates a so-called progressive growing of the input which is seen at the top of Fig.”
ICASSP 2023 Deep Noise Suppression Challenge (2023, methodology): “…Personalized P.835 framework is improved and now includes voice recognition, robust spam ﬁltering and more accurate evaluation of enhanced testclips with noise and neighboring talkers; (iv) Personalized P.835 framework uses cleaned enrollment speech which was enhanced using a non-causal model.”
Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement (2021, methodology): “Hu et al. [3] built upon the DCUNET and introduced a deep complex convolution recurrent network ar X iv :2 10 6.”
Personalized speech enhancement: new models and Comprehensive evaluation (2021, background): “Deep neural networks (DNNs) have shown promising results compared with the traditional signal processing methods for single-channel speech enhancement (SE) [2, 3, 4, 5].”
Phase-aware Single-stage Speech Denoising and Dereverberation with U-Net (2020, methodology): “To better design the time-domain cos similarity loss function proposed in [6], we change it into a multi-scale version of it with proper emphasis functions and show the effectiveness of it.”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, background): “In recent years, various deep learning-based models have been developed for SE (Lu et al., 2013; Xu et al., 2014; Wang et al., 2014; Xu et al., 2015; Ochiai et al., 2017; Luo & Mesgarani, 2018; Grais et al., 2018; Germain et al., 2018; Chai et al., 2018; Choi et al., 2019).”
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (2020, methodology): “We design the complex Conv2d block according to that in DCUNET [25].”