Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder

By Cristina Garbacea, Aäron van den Oord, Yazhe Li, Felicia S. C. Lim, Alejandro Luebs, Oriol Vinyals, Thomas C. Walters. 2019

140 citations recorded by the source corpus.

In order to efficiently transmit and store speech signals, speech codecs create a minimally redundant representation of the input signal which is then decoded at the receiver with the best possible perceptual quality. In this work we demonstrate that a neural network architecture based on VQ-VAE with a WaveNet decoder can be used to perform very low bit-rate speech coding with high reconstruction quality. A prosody-transparent and speaker-independent model trained on the LibriSpeech corpus coding audio at 1.6 kbps exhibits perceptual quality which is around halfway between the MELP codec at 2.4 kbps and AMR-WB codec at 23.05 kbps. In addition, when training on high-quality recorded speech with the test speaker included in the training set, a model coding speech at 1.6 kbps produces output of similar perceptual quality to that generated by AMR-WB at 23.05 kbps.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “A VQ-VAE speech codec was proposed in [6], operating at 1 .”
High Fidelity Neural Audio Compression (2022, methodology): “Gârbacea et al. (2019) conditioned a WaveNet based model on discrete units obtained from a VQ-VAE (Van Den Oord et al., 2017; Razavi et al., 2019) model, while Skoglund & Valin (2019) tried feeding the Opus codec (Valin et al., 2012) to a WaveNet to further improve its perceptual quality.”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Furthermore, recent advancements include softmax quantization (Kankanahalli, 2018), exponential moving average (EMA) (Gârbacea et al., 2019), and Gumble-softmax (Yang et al., 2022) also work well for optimization of the VQ module.”
High-Fidelity Audio Compression with Improved RVQGAN (2023, neutral): “The first VQ-VAE based speech codec was proposed in [13] operating at 1.”
Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, background): “For example, the neural codecs with softmax quantization [21], straight-through gradient, exponential moving average (EMA) [22, 23], and Gumble-softmax [31] have been recently proposed and work well for gradient propagation.”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “This model encodes the raw audio into a sequence of discrete tokens from 256 possible tokens [34] with a hop size of 160 raw audio samples.”
Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, methodology): “Fully learned codecs like [7, 8] open new avenues for learning based compression and demonstrate strong results for compressing clean speech.”
End-to-End Neural Speech Coding for Real-Time Communications (2022, neutral): “Existing neural audio codecs could mainly be grouped into two categories, generative de-coder models [1, 2, 3, 4, 5, 6] and end-to-end neural audio coding [7, 8, 9, 10].”
Improving Opus Low Bit Rate Quality with Neural Speech Synthesis (2019, methodology): “Since then, WaveNet has been used for parametric coding that significantly out-perform more traditional vocoders, either using an existing vocoder bit stream [8], or with a trained quantized feature set [9].”
A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet (2019, methodology): “Although other work has demonstrated similar or higher speech quality at similar bitrates [8, 9, 10], we believe this is the first neural vocoder that can operate in real-time on general-purpose hardware and mobile devices.”
Robust Low Rate Speech Coding Based on Cloned Networks and Wavenet (2020, methodology): “[21] combines WaveNet with VQ-VAE [22], operates at 1.”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “The low-bit rate representation is typically a discrete representation of human speech learned through autoencoding [19, 26, 28, 38] or speech features obtained through self-supervised training [44].”
Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, background): “On the other hand, some researchers propose end-to-end neural networks based on the vector-quantized variational autoencoder framework (VQ-VAE [9]) where the encoder, codebook and the decoder are learned in a joint fashion [10, 11, 12, 13, 14, 15].”
Dynamical Variational Autoencoders: A Comprehensive Review (2020, methodology): “Other examples include VQ-VAE (that is a mix of VAE and vector quantization of z) applied to speech coding (van den Oord et al., 2017; Gârbacea et al., 2019), and video compression with rate-distortion autoencoders (Habibian et al., 2019).”
Source Coding of Audio Signals with a Generative Model (2020, methodology): “Deep generative models have been successfully used for speech coding [1–5], providing a significant improvement to the perceptual quality–bitrate tradeoff.”
Speech Bandwidth Extension with Wavenet (2019, methodology): “In addition, the WaveNet architecture has been used conditioned on log-mel spectrograms for text-to-speech [2] and other low-dimensional latent representations for speech coding [3, 4].”