End-To-End Optimized Speech Coding with Deep Neural Networks

By Srihari Kankanahalli. 2017

74 citations recorded by the source corpus.

Modern compression algorithms are often the result of laborious domain-specific research; industry standards such as MP3, JPEG, and AMR-WB took years to develop and were largely hand-designed. We present a deep neural network model which optimizes all the steps of a wideband speech coding pipeline (compression, quantization, entropy coding, and decompression) end-to-end directly from raw speech data - no manual feature engineering necessary, and it trains in hours. In testing, our DNN-based coder performs on par with the AMR -WB standard at a variety of bitrates (~9kbps up to ~24kbps). It also runs in realtime on a 3.8GhZ Intel CPU.

Subfields: Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “More recently, a more sophisticated deep convolutional network for speech compression was described in [38].”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Furthermore, recent advancements include softmax quantization (Kankanahalli, 2018), exponential moving average (EMA) (Gârbacea et al., 2019), and Gumble-softmax (Yang et al., 2022) also work well for optimization of the VQ module.”
Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, background): “For example, the neural codecs with softmax quantization [21], straight-through gradient, exponential moving average (EMA) [22, 23], and Gumble-softmax [31] have been recently proposed and work well for gradient propagation.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “Moreover, combining such highquality synthesis systems with differentiable quantization [22]– [24], allows training end-to-end neural codecs [16], [25]–[27] by compressing activations in a bottleneck layer.”
Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding (2019, methodology): “Therefore, for the subjective quality, we perform MUSHRA tests [31] to show that CMRL with an LPC residual input works better than AMR-WB and OPUS at high bitrates.”
Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder (2019, methodology): “In the speech domain, end-to-end coding operating at rates as low as 369 bps by use of extracted acoustic features [5] and an end-to-end optimized wideband codec performing on par with the adaptive multirate wideband (AMR-WB) codec at 9-24 kbps [6] have both recently been proposed.”
Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding (2020, methodology): “auditory system does, L2(y||ŷ) = ∑N i=1 ∑L l=1(y (i) l − ŷ l )(2), where y stands for a mel spectrum with L frequency subbands as proposed in [21].”
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis (2022, background): “The low-bit rate representation is typically a discrete representation of human speech learned through autoencoding [19, 26, 28, 38] or speech features obtained through self-supervised training [44].”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “In recent years, DNN aided compression algorithms have achieved significant results, often outperforming state-of-theart standardised codecs in a variety of source domains, from image [156]–[160], to video [161]–[169], speech [170], [171], and audio [172], [173] compression.”
Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, background): “On the other hand, some researchers propose end-to-end neural networks based on the vector-quantized variational autoencoder framework (VQ-VAE [9]) where the encoder, codebook and the decoder are learned in a joint fashion [10, 11, 12, 13, 14, 15].”
Deep Generative Models for Distribution-Preserving Lossy Compression (2018, methodology): “These methods were shown to outperform traditional compression standards in image compression [1–10], speech compression [11], and video compression [12] under several distortion measures.”
Harp-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding (2021, unknown): “Following the same method proposed in [5], we first let the network train without quantization for about 8 epochs, then introduce the soft-to-hard quantization module.”
Disentangling speech from surroundings in a neural audio codec (2022, background): “Recent progress in generative models of audio have allowed for the development of neural codecs [1, 2, 3, 4] that are competitive with traditional codecs.”
Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding (2022, unknown): “End-to-end neural codecs that train the encoder, quantizer, and decoder jointly work as an alternative to the low-bitrate generative speech vocoders [14, 15, 16, 17].”
Machine Learning in the Air (2019, background): “all these data domains [41]–[44], with results meeting or surpassing state-of-the-art expert-based compression techniques,”
End-to-End Neural Audio Coding in the MDCT Domain (2023, background): “The overall modules are jointly optimized in an end-to-end manner to resolve the rate-distortion trade-off [8], which leads to better performance than the conventional rule-based codecs [11–18].”