Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec

By Yi-Chiao Wu, Israel Dejene Gebru, Dejan Markovic, Alexander Richard. 2023

139 citations recorded by the source corpus.

A good audio codec for live applications such as telecommunication is characterized by three key properties: (1) compression, i.e. the bitrate that is required to transmit the signal should be as low as possible; (2) latency, i.e. encoding and decoding the signal needs to be fast enough to enable communication without or with only minimal noticeable delay; and (3) reconstruction quality of the signal. In this work, we propose an open-source, streamable, and real-time neural audio codec that achieves strong performance along all three axes: it can reconstruct highly natural sounding 48 kHz speech signals while operating at only 12 kbps and running with less than 6 ms (GPU)/10 ms (CPU) latency. An efficient training paradigm is also demonstrated for developing such neural audio codecs for real-world scenarios. Both objective and subjective evaluations using the VCTK corpus are provided. To sum up, AudioDec is a well-developed plug-and-play benchmark for audio codec applications.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Shujie Liu
5. Yanqing Liu
6. Tao Qin
7. Tie-Yan Liu
8. Furu Wei

In Model Architectures

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Shujie Liu
5. Jinyu Li
6. Shiliang Zhang
7. Sanyuan Chen
8. Long Zhou

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Furu Wei
8. Lei He

In Signal Processing

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Yanqing Liu
5. Shujie Liu
6. Sanyuan Chen
7. Lei He
8. Tao Qin

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “The new codecs include Vocos [62], SpeechTokenizer [63], AudioDec [64], AcademiCodec [65], Descript-audio-codec (DAC) [66], Fun-Codec [67], and RepCodec [68].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “These include Vocos [Siuzdak, 2023], SpeechTokenizer [Zhang et al., 2023a], AudioDec [Wu et al., 2023], AcademiCodec [Yang et al., 2023a], Descript-audio-codec (DAC) [Kumar et al., 2024], FunCodec [Du et al., 2024b], and RepCodec [Huang et al., 2023b].”
• FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, background): “: descriptinc/descript-audio-codec [20], • AudioDec: facebookresearch/AudioDec [19].”
• RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Discrete speech tokens can be divided into two categories: Semantic tokens (Lee et al., 2022b; Hsu et al., 2021; Chung et al., 2021) and Acoustic tokens (Zeghidour et al., 2022; Défossez et al., 2022; Wu et al., 2023).”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, unknown): “Moreover, the quality of speech tokenizer is essential for AR models to achieve high-fidelity synthesis [20, 21, 22, 23, 24, 25, 26].”