Real-Time Speech Frequency Bandwidth Extension

By Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu, Dominik Roblek. 2020

71 citations recorded by the source corpus.

In this paper we propose a lightweight model for frequency bandwidth extension of speech signals, increasing the sampling frequency from 8kHz to 16kHz while restoring the high frequency content to a level almost indistinguishable from the 16kHz ground truth. The model architecture is based on SEANet (Sound EnhAncement Network), a wave-to-wave fully convolutional model, which uses a combination of feature losses and adversarial losses to reconstruct an enhanced version of the input speech. In addition, we propose a variant of SEANet that can be deployed on-device in streaming mode, achieving an architectural latency of 16ms. When profiled on a single core of a mobile CPU, processing one 16ms frame takes only 1.5ms. The low latency makes it viable for bi-directional voice communication systems.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “To train such a model, we modify Algorithm 1 in the following way: for each input example, we sample n q uniformly at random in [1; N q ] and only use quantizers Q i for i = 1 . . . n q .”
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “While SoundStream utilizes streaming SEANet [7,8] as its encoder and decoder, Encodec incorporates extra LSTM [9] layers and a Demos are available at: https://funcodec.github.io Transformer-based language model [10] to improve the sequence modeling ability.”
AudioGen: Textually Guided Audio Generation (2022, neutral): “2021).”
Universal Speech Enhancement with Score-based Diffusion (2022, unknown): “…Mao, 2022) 2.92 3.52 PoCoNet (Isik et al., 2020) 2.75 3.42 FullSubNet (Hao et al., 2021) 3.31 2.78 0.96 DCCRN+ (Lv et al., 2021) 2.84 3.33 CTS-Net (Li et al., 2021a) 2.92 3.59 3.42 2.94 0.97 Demucs (Défossez et al., 2020) 3.07 3.63 0.95 SN-Net (Zheng et al., 2021) 3.12 3.60 3.39 SE-Conformer (Kim…”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “It is noteworthy that we signiﬁcantly outperform the SEANet (Tagliasacchi et al., 2020) model, which is trained in a similar adversarial manner and has a larger number of parameters, but does not take into account spectral information.”