Speech Denoising with Deep Feature Losses

By François G. Germain, Qifeng Chen, Vladlen Koltun. 2018

169 citations recorded by the source corpus.

We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly. Given input audio containing speech corrupted by an additive background signal, the system aims to produce a processed signal that contains only the speech content. Recent approaches have shown promising results using various deep network architectures. In this paper, we propose to train a fully-convolutional context aggregation network using a deep feature loss. That loss is based on comparing the internal feature activations in a different network, trained for acoustic environment detection and domestic audio tagging. Our approach outperforms the state-of-the-art in objective speech quality metrics and in large-scale perceptual experiments with human listeners. It also outperforms an identical network trained using traditional regression losses. The advantage of the new approach is particularly pronounced for the hardest data with the most intrusive background noise, for which denoising is most needed and most challenging.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “We further compare our Streaming SEANet with two base-lines, the original ofﬂine SEANet and the model by Germain et al. [9], both of which are more than an order of magnitude more expensive in computational cost, with architectural latency in the hundreds of milliseconds.”
Real Time Speech Enhancement in the Waveform Domain (2020, background): “Over the last years, there has been a growing interest towards DNN based methods for speech enhancement [34, 35, 36, 37, 20, 38, 39, 22, 40, 7, 41, 8, 42, 21, 37].”
Bandwidth Extension on Raw Audio via Generative Adversarial Networks (2019, methodology): “The method from [11] uses a VGG-based [36] network as a feature loss for speech denoising, and trains the loss network on classification and audio tagging tasks from DCASE 2016 [29].”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, background): “Finally, Deep Feature Loss (Germain et al., 2018) also operates on the raw waveform and is trained with a deep feature loss from another network that classiﬁes acoustic environments.”
Expressive TTS Training With Frame and Style Reconstruction Loss (2020, methodology): “In speech enhancement, perceptual loss has been used successfully in end-to-end speech denoising pipeline, with an auxiliary network pre-trained on audio classification task [73].”
PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network (2019, methodology): “Firstly, our method has very large gain over time-domain methods like SEGAN (Pascual, Bonafonte, and Serra 2017), Wavenet (Rethage, Pons, and Serra 2018), and DFL (Germain, Chen, and Koltun 2018) on all the ﬁve metrics, even though these time-domain methods are free of phase-prediction problem.”
Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, methodology): “As shown in Table 1, STFT-TCN outperforms Conv-TasNet, DFL [13], and MDPhD [33] in terms of all perceptual quality metrics.”
Upsampling Artifacts in Neural Audio Synthesis (2020, background): “Issue (iii): Loss Function — Loss-related tonal artifacts can appear when using adversarial losses [1] and/or deep feature losses [16], since using CNNs as a loss involves a transposed convolution in the backpropagation step that can cause high-frequency patterns in the gradient [15].”
Weighted Speech Distortion Losses for Neural-Network-Based Real-Time Speech Enhancement (2020, background): “Methods along this line of thought include learning multiple objectives from heterogeneous features [16, 17, 18], jointly optimizing the final goal and its sub-targets (e.”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “Each of the corresponding time-frequency (t, f) representations computed by STFT is denoted as Xt,f ∈ C, Yt,f ∈ C, Zt,f ∈ C, and Ŷt,f ∈ C.”
PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss (2020, methodology): “Recent phase-aware models use a complex ratio mask instead of magnitude gain [6, 7], while other approaches work directly in the waveform domain [8, 9, 10].”