LMCodec: A Low Bitrate Speech Codec with Causal Transformer Models

By Teerapat Jenrungrot, Michael Chinen, W. Kleijn, Jan Skoglund, Zalán Borsos, Neil Zeghidour, Marco Tagliasacchi. 2023

31 citations recorded by the source corpus.

We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates. Example audio is available at https://mjenrungrot.github.io/chrome-media-audio-papers/publications/lmcodec.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiliang Zhang
2. Zhihao Du
3. Ziyang Ma
4. Xiaohuan Zhou
5. Zhijie Yan
6. Kai Hu
7. Siqi Zheng
8. Yunfei Chu

In Signal Processing

1. Zhihao Du
2. Shiliang Zhang
3. Ziyang Ma
4. Zhijie Yan
5. Kai Hu
6. Siqi Zheng

Citations in the corpus

• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “Moreover, the indices of the remaining codec groups are challenging to predict due to the multi-modal distribution nature of codec tokens (Jenrungrot et al., 2023).”