Wavenet Based Low Rate Speech Coding

By W. Kleijn, Felicia S. C. Lim, Alejandro Luebs, Jan Skoglund, Florian Stimberg, Quan Wang, Thomas C. Walters. 2017

154 citations recorded by the source corpus.

Traditional parametric coding of speech facilitates low rate but provides poor reconstruction quality because of the inadequacy of the model used. We describe how a WaveNet generative speech model can be used to generate high quality speech from the bit stream of a standard parametric coder operating at 2.4 kb/s. We compare this parametric coder with a waveform coder based on the same generative model and show that approximating the signal waveform incurs a large rate penalty. Our experiments confirm the high performance of the WaveNet based coder and show that the speech produced by the system is able to additionally perform implicit bandwidth extension and does not significantly impair recognition of the original speaker for the human listener, even when that speaker has not been used during the training of the generative model.

Subfields: Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “For example, WaveNet [4], a strong generative model originally applied to generate speech from text, was adopted as a decoder in a neural codec [5], [6].”
High Fidelity Neural Audio Compression (2022, neutral): “Neural based audio codecs have been recently proposed and demonstrated promising results (Kleijn et al., 2018; Valin & Skoglund, 2019b; Lim et al., 2020; Kleijn et al., 2021; Zeghidour et al., 2021; Omran et al., 2022; Lin et al., 2022; Jayashankar et al., 2022; Li et al.; Jiang et al., 2022),…”
Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “The second category is the vocoder-based codec, which directly reconstructs audio signals using neural vocoders conditioned on the codes from other parametric coders [16] or quantized acoustic features [17–19].”
HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “Recently, researchers have proposed several neural network-based audio codecs that show promising results [11, 12, 21–24].”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, background): “Neural speech codecs have been recently proposed and demonstrated promising results [30, 31].”
Generative Speech Coding with Predictive Variance Regularization (2021, background): “Generative synthesis of a wide range of unknown voices with a single model results in only a minor reduction of speaker identiﬁability [7].”
Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, background): “Recently, there have been successful efforts in building learned codecs; starting with replacing the decoders with learned decoders for ﬁxed encoders, which can operate as low as 2:4 kb/s to 1:6 kb/s [2, 3]. These learned decoders leverage advances in speech synthesizing generative models such as Wa”
ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric (2020, methodology): “One example is the WaveNet-based low bitrate coder [10], which is generative in nature.”
Audio Codec Enhancement with Generative Adversarial Networks (2020, background): “In fact, it was shown in [29] that the enhanced quality achieved with a generative decoder was not predicted by the objective tool.”
Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, background): “Recent DNN-based codecs [13][14][15][16] model speech signals in time domain directly without T-F transformation.”
Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder (2019, methodology): “[9] use a learned WaveNet decoder to produce audio comparable in quality to that produced by the AMR-WB [10] codec at 23.”
LPCNET: Improving Neural Speech Synthesis through Linear Prediction (2018, methodology): “The neural networks in many neural speech synthesis approaches [4, 5, 6, 7, 15] have to model the entire speech production process, including glottal pulses, noise excitation, as well as the response of the vocal tract.”
It's Raw! Audio Generation with State-Space Models (2022, methodology): “With AR models, computing the exact likelihood is tractable, which makes them simple to train, and lends them to applications such as lossless compression [22] and posterior sampling [19].”
End-to-End Neural Speech Coding for Real-Time Communications (2022, neutral): “Existing neural audio codecs could mainly be grouped into two categories, generative de-coder models [1, 2, 3, 4, 5, 6] and end-to-end neural audio coding [7, 8, 9, 10].”
Improving Opus Low Bit Rate Quality with Neural Speech Synthesis (2019, methodology): “Since then, WaveNet has been used for parametric coding that significantly out-perform more traditional vocoders, either using an existing vocoder bit stream [10], or with a quantized”
A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet (2019, methodology): “Although other work has demonstrated similar or higher speech quality at similar bitrates [8, 9, 10], we believe this is the first neural vocoder that can operate in real-time on general-purpose hardware and mobile devices.”
High-quality Speech Coding with Sample RNN (2018, methodology): “In order to facilitate comparison to [12] we carried our experiments with the same multi-speaker dataset, namely the Wall Street Journal set WSJ0 [17].”
Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages (2020, methodology): “Tacotron-2 uses WaveNet [16] as a vocoder conditioned on the mel-spectogram instead of using the Grifﬁn-Lim algorithm as in Tacotron.”