A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement

By Jean-Marc Valin. 2017

247 citations recorded by the source corpus.

Despite noise suppression being a mature area in signal processing, it remains highly dependent on fine tuning of estimator algorithms and parameters. In this paper, we demonstrate a hybrid DSP/deep learning approach to noise suppression. We focus strongly on keeping the complexity as low as possible, while still achieving high-quality enhanced speech. A deep recurrent neural network with four hidden layers is used to estimate ideal critical band gains, while a more traditional pitch filter attenuates noise between pitch harmonics. The approach achieves significantly higher quality than a traditional minimum mean squared error spectral estimator, while keeping the complexity low enough for real-time operation at 48 kHz on a low-power CPU.

Subfields: Computational Mechanics, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Speech & Audio

In Signal Processing

Citations in the corpus

TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, methodology): “RNNoise is based on Recurrent Neural Networks; more specifically Gated Recurrent Unit [45], and demonstrated good performance for noise suppression.”
Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, methodology): “We also experiment with the combination of a speech enhancement model (RNNoise) [17] and the compression only model.”
LPCNET: Improving Neural Speech Synthesis through Linear Prediction (2018, methodology): “The cepstrum uses the same band layout as [19] and the pitch estimator is based on an open-loop cross-correlation search.”
Improving Opus Low Bit Rate Quality with Neural Speech Synthesis (2019, methodology): “s. To make both models more robust to variations in the input, we augment the training data. The signal level is varied over a 40 dB range and the frequency response is varied according to eq. (7) in [19]). 5. EVALUATION The source code for the LPCNet model is available at https://github.com/mozilla”
A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet (2019, background): “from 18 Bark-spaced bands following the same layout as [17].”
Harmonic Gated Compensation Network Plus for ICASSP 2022 DNS Challenge (2022, background): “[3] explores an SE model combining signal processing and deep learning, but simple structure limits its performance.”
A scalable noisy speech dataset and online subjective test framework (2019, methodology): “For denoising, the loss function is the mean squared error between the ideal and predicted per-band gain function raised to an exponent as defined in [1] eq 8.”
Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement (2020, methodology): “The Mel-spectrogram denoise masks is extracted by a speech enhancement model which accepts Mel-spectrogram as input, the model structure is a variant of the CNN-RNN-FC structure, and it is very similar to the model in [18].”
Weighted Speech Distortion Losses for Neural-Network-Based Real-Time Speech Enhancement (2020, methodology): “Not only did all our selected systems significantly outperform the improved RNNoise (RNNoiseI) trained on MSE presented in [22], we were surprised that the listening test subjects preferred a rather low setting for the speech distortion weight α.”
ICASSP 2023 Deep Noise Suppression Challenge (2023, methodology): “We introduced the following changes in this challenge: (i) there are two tracks: Headset, and Speakerphone both has desktop and mobile recordings in testsets; (ii) All testclips in both tracks has 10-30s enrollment speech (primary talker) with or without noise; (iii) Personalized P.835 framework is…”
Data Augmentation and Loss Normalization for Deep Noise Suppression (2020, methodology): “in [6–10].”
End-To-End Speech Synthesis Applied to Brazilian Portuguese (2020, unknown): “We used the library RNNoise [Valin, 2017].”
PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss (2020, methodology): “Synthetic without Reverb Synthetic with Reverb Methods PESQ CBAK COVL CSIG PESQ CSIG CBAK COVL RNNoise [24] 1.”
FB-MSTCN: A Full-Band Single-Channel Speech Enhancement Method Based on Multi-Scale Temporal Convolutional Network (2022, background): “In the full-band real-time scenarios, modeling sub-band spectral features, such as RNNoise [10] and PercepNet [11], is an effective strategy to balance the computational complexity and speech denoising performance.”
LMCodec: A Low Bitrate Speech Codec with Causal Transformer Models (2023, methodology): “We use an open-source RNNoise model [24], which uses Mel-Frequency Cepstral Coefficients (MFCC) and outputs the probability of voice activity every 10ms frame size.”