Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding

By Kai Zhen, Jongmo Sung, Mi Suk Lee, Seungkwon Beack, Minje Kim. 2019

43 citations recorded by the source corpus.

Speech codecs learn compact representations of speech signals to facilitate data transmission. Many recent deep neural network (DNN) based end-to-end speech codecs achieve low bitrates and high perceptual quality at the cost of model complexity. We propose a cross-module residual learning (CMRL) pipeline as a module carrier with each module reconstructing the residual from its preceding modules. CMRL differs from other DNN-based speech codecs, in that rather than modeling speech compression problem in a single large neural network, it optimizes a series of less-complicated modules in a two-phase training scheme. The proposed method shows better objective performance than AMR-WB and the state-of-the-art DNN-based speech codec with a similar network architecture. As an end-to-end model, it takes raw PCM signals as an input, but is also compatible with linear predictive coding (LPC), showing better subjective quality at high bitrates than AMR-WB and OPUS. The gain is achieved by using only 0.9 million trainable parameters, a significantly less complex architecture than the other DNN-based codecs in the literature.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “An end-to-end audio codec targeting general audio at high bitrates (i.e., above 64 kbps) was proposed in [40].”
Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “Therefore, scalable codecs decomposing the fine-coarse structures of the encoded latent codes using residual VQ [25], multi-scale VQ [26], and crossscale VQ [27] or the output waveforms using cross-module residual learning [24] are introduced for tractable hierarchical codebooks.”
End-to-End Neural Speech Coding for Real-Time Communications (2022, neutral): “Existing neural audio codecs could mainly be grouped into two categories, generative de-coder models [1, 2, 3, 4, 5, 6] and end-to-end neural audio coding [7, 8, 9, 10].”
Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, background): “On the other hand, some researchers propose end-to-end neural networks based on the vector-quantized variational autoencoder framework (VQ-VAE [9]) where the encoder, codebook and the decoder are learned in a joint fashion [10, 11, 12, 13, 14, 15].”
Harp-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding (2021, unknown): “In [12], the idea of cascading multiple autoencoders [13] also show performance improvement, while the concatenation linearly increases model complexity.”
Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding (2020, methodology): “2) Competing Models: We consider two baseline models: Model-A trained by the SSE (L1) and Model-B also learns on the mel-frequency loss (L2) as in recent neural speech codecs [21], [26], [31].”
Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding (2022, unknown): “End-to-end neural codecs that train the encoder, quantizer, and decoder jointly work as an alternative to the low-bitrate generative speech vocoders [14, 15, 16, 17].”