Differentiable Consistency Constraints for Improved Deep Speech Enhancement

By Scott Wisdom, John R. Hershey, Kevin Wilson, Jeremy Thorpe, Michael Chinen, Brian Patton, Rif A. Saurous. 2018

131 citations recorded by the source corpus.

In recent years, deep networks have led to dramatic improvements in speech enhancement by framing it as a data-driven pattern recognition problem. In many modern enhancement systems, large amounts of data are used to train a deep network to estimate masks for complex-valued short-time Fourier transforms (STFTs) to suppress noise and preserve speech. However, current masking approaches often neglect two important constraints: STFT consistency and mixture consistency. Without STFT consistency, the system’s output is not necessarily the STFT of a time-domain signal, and without mixture consistency, the sum of the estimated sources does not necessarily equal the input mixture. Furthermore, the only previous approaches that apply mixture consistency use real-valued masks; mixture consistency has been ignored for complex-valued masks.

Subfields: Artificial Intelligence, Computational Mechanics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Signal Processing

Citations in the corpus

SEANet: A Multi-modal Speech Enhancement Network (2020, neutral): “More recently, deep architectures have been adopted to improve speech enhancement based on spectral masking [10].”
End-to-End Neural Speech Coding for Real-Time Communications (2022, positive): “To ensure STFT consistency [16], the decoded spectrum is ﬁrst transformed into waveform domain through an inverse STFT and then transformed into TF-domain again through a STFT to calculate the loss.”
Performance Study of a Convolutional Time-Domain Audio Separation Network for Real-Time Speech Denoising (2020, background): “Recent advances in deep learning systems for speech separation have shown to be effective in solving this problem [5, 6, 7, 8, 9].”
Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, methodology): “To satisfy STFT consistency [9], we apply the STFT to estimated signals Ŝk and clean signals Sk, which are created by dividing, overlapping and concatenating ŝk and sk , as ŴSPEC,k = USTFTŜk,WSPEC,k = USTFTSk.”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “Note that similar techniques have been used in speech enhancement literature as a standalone solution (Wisdom et al., 2019; Jansson et al., 2017).”
Improving Sound Event Detection in Domestic Environments using Sound Separation (2020, methodology): “A weighted mixture consistency projection layer [24] is applied to the separated waveforms to be consistent with the input mixture waveform where the per-source weights are predicted by an additional dense layer using the penultimate output of TDCN++.”
Self-Supervised Learning from Automatically Separated Sound Scenes (2021, background): “Separated sources are constrained to add up to the input mixture via a consistency projection layer [29].”
Data Augmentation and Loss Normalization for Deep Noise Suppression (2020, methodology): “ain. 3 Network and Training We use a rather straightforward recurrent network architecture based on gated recurrent units (GRUs) [11] and feed forward (FF) layers, similar to the core architecture of [12] without convolutional encoder layers. Input features are the logarithmic power spectrum P= log ”
What’s all the Fuss about Free Universal Sound Separation Data? (2020, methodology): “…with complex spectrogram input, and initial separated sources s are produced by applying an inverse STFT. Finally, a mixture consistency layer [29] is applied to these initial separated source waveforms: which projects the separated sources such that they sum up to the original input mixture…”
Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, methodology): “To keep STFT consistency [24], the reconstructed spectrum is ﬁrst transformed to time domain and then to the frequency domain to calculate the loss.”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, methodology): “An inverse STFT and forward STFT are conducted on speech and noise branches before calculating the loss to ensure STFT consistency as that in (Wisdom et al. 2019).”