Generative Speech Coding with Predictive Variance Regularization

By W. Kleijn, Andrew Storus, Michael Chinen, Tom Denton, Felicia S. C. Lim, Alejandro Luebs, Jan Skoglund, Hengchin Yeh. 2021

79 citations recorded by the source corpus.

The recent emergence of machine-learning based generative models for speech suggests a significant reduction in bit rate for speech codecs is possible. However, the performance of generative models deteriorates significantly with the distortions present in real-world input signals. We argue that this deterioration is due to the sensitivity of the maximum likelihood criterion to outliers and the ineffectiveness of modeling a sum of independent signals with a single autoregressive model. We introduce predictive-variance regularization to reduce the sensitivity to outliers, resulting in a significant increase in performance. We show that noise reduction to remove unwanted signals can significantly increase performance. We provide extensive subjective performance evaluations that show that our system based on generative modeling provides state-of-the-art coding performance at 3 kb/s for real-world speech signals at reasonable computational complexity.

Subfields: Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Speech & Audio

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “For the lowest bitrates, we also compare the performance of the recently presented Lyra codec [8] which is an autoregressive generative codec operating at 3 kbps.”
High Fidelity Neural Audio Compression (2022, neutral): “…based audio codecs have been recently proposed and demonstrated promising results (Kleijn et al., 2018; Valin & Skoglund, 2019b; Lim et al., 2020; Kleijn et al., 2021; Zeghidour et al., 2021; Omran et al., 2022; Lin et al., 2022; Jayashankar et al., 2022; Li et al.; Jiang et al., 2022), where…”
End-to-End Neural Speech Coding for Real-Time Communications (2022, neutral): “Existing neural audio codecs could mainly be grouped into two categories, generative de-coder models [1, 2, 3, 4, 5, 6] and end-to-end neural audio coding [7, 8, 9, 10].”
Taming Visually Guided Sound Generation (2021, methodology): “We provide a qualitative comparison of reconstructions of Lyra [33] (only speech), SoundStream (only speech and music), and Spectrogram VQGAN (open-domain) in the supplementary and our project page.”
Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding (2022, methodology): “Various other methods have been developed to improve the quality of the features, regarding robustness [18, 19], scalability [20] and the variability issues [21].”
Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, methodology): “…codec that is widely used for real-time communications, supporting narrowband to fullband speech and audio with bitrate from 6kbps to 510kbps. Lyra [6] is an autoregressive generative speech codec proposed recently, which reconstructs high quality speech at 3 kbps. Low bitrate scenarios Figure…”
Variable-rate discrete representation learning (2021, background): “16-bit 24 kHz mono audio 384,000 8-bit 24 kHz mono audio (mu-law) 192,000 SoTA speech codec (Kleijn et al., 2021) 3,000 VQ-VAE code sequence 500 SlowAE code sequence (LibriTTS, ∼ 75Hz) 893 SlowAE code sequence (Libri-light, ∼ 51Hz) 607 VQ-VAE entropy bound ≤291 SlowAE entropy bound (LibriTTS, ∼ 75Hz”
A Streamwise Gan Vocoder for Wideband Speech Coding at Very Low Bit Rate (2021, unknown): “More recent work [14] presented a new neural speech decoder (Lyra) compressing speech at 3 kbit / s .”
Disentangled Feature Learning for Real-Time Neural Speech Coding (2022, background): “They are either based on strong generative models for decoding [1–3] or end-to-end learning by the VQ-VAE [8] framework [4–6].”