Raw waveform-based speech enhancement by fully convolutional networks

By Szu-Wei Fu, Yu Tsao, Xugang Lu, Hisashi Kawai. 2017

211 citations recorded by the source corpus.

This study proposes a fully convolutional network (FCN) model for raw waveform-based speech enhancement. The proposed system performs speech enhancement in an end-to-end (i.e., waveform-in and waveform-out) manner, which differs from most existing denoising methods that process the magnitude spectrum (e.g., log power spectrum (LPS)) only. Because the fully connected layers, which are involved in deep neural networks (DNN) and convolutional neural net-works (CNN), may not accurately characterize the local in-formation of speech signals, particularly with high frequency components, we employed fully convolutional layers to model the waveform. More specifically, FCN consists of only convolutional layers and thus the local temporal structures of speech signals can be efficiently and effectively preserved with relatively few weights. Experimental results show that DNN- and CNN-based models have limited capability to restore high frequency components of waveforms, thus leading to decreased intelligibility of enhanced speech. By contrast, the proposed FCN model can not only effectively recover the waveforms but also outperform the LPS- based DNN baseline in terms of short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ). In addition, the number of model parameters in FCN is approximately only 0.2% com-pared with that in both DNN and CNN.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

Speech Denoising with Deep Feature Losses (2018, methodology): “sults in speech processing [19, 20] including speech denoising [21, 22, 23, 24].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Another recent trend is to avoid the phase inconsistency problem altogether by operating in the time domain, using convolutional neural networks [30, 31], WaveNet [32], generative adversarial networks [33], or encoder-decoder architectures [34].”
Noise Reduction in ECG Signals Using Fully Convolutional Denoising Autoencoders (2019, background): “Fu et al. pointed out that for speech signal processing, this interdependency may make DNN improper to model waveforms because fully connected layers average out the relation between neighboring samples [26].”
Interspeech 2021 Deep Noise Suppression Challenge (2021, background): “In recent years, learning-based approaches have shown promising results [3, 4, 5].”
Speech Enhancement and Dereverberation With Diffusion-Based Generative Models (2022, background): “frequency (T-F) masking [4], spectral mapping [5], or operating directly in the time domain [6].”
Supervised Speech Separation Based on Deep Learning: An Overview (2017, unknown): “[47] developed a fully convolutional network (a CNN 782 with fully connected layers removed) for speech enhancement.”
A New Framework for CNN-Based Speech Enhancement in the Time Domain (2019, unknown): “In [9], authors explore CNNs for speech enhancement and claim that fully connected layers inside a DNN are not suitable for the time domain enhancement”
Generative Adversarial Network-Based Glottal Waveform Model for Statistical Parametric Speech Synthesis (2017, methodology): “[26, 28] that convolutional Parametrisation with QCP Speech signal”
TCNN: Temporal Convolutional Neural Network for Real-time Speech Enhancement in the Time Domain (2019, unknown): “In [9], the authors demonstrate the effectiveness of fully convolutional neural networks for time domain speech enhancement.”
ICASSP 2021 Deep Noise Suppression Challenge (2020, background): “In recent years, learning-based approaches have shown promising results [3, 4, 5].”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, methodology): “Some methods operate in the time domain by estimating speech signals from raw-waveform noisy signals in an end-to-end way (Fu et al. 2017; Pascual, Bonafonte, and Serra 2017; Pandey and Wang 2019).”
Convolutional Neural Networks to Enhance Coded Speech (2018, background): “, log-power spectrum [32]–[34] or complex spectrogram [35], or a mapping directly between the noisy raw speech waveform and clean raw speech waveform [36], [37].”