High-Fidelity Audio Compression with Improved RVQGAN

By Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, I. Kumar, Kundan Kumar. 2023

803 citations recorded by the source corpus.

Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Tao Qin
4. Sheng Zhao
5. Tie-Yan Liu
6. Shujie Liu
7. Furu Wei
8. Yanqing Liu

In Computer Vision and Pattern Recognition

1. Shiliang Zhang
2. Shujie Liu
3. Furu Wei
4. Xiang-Yang Li
5. Kaitao Song
6. Xu Tan
7. Jinyu Li
8. Sanyuan Chen

In Generative Models

1. Yichong Leng
2. Jinyu Li
3. Jiang Bian
4. Zeqian Ju
5. Chenpeng Du
6. Dongchao Yang
7. Shiliang Zhang
8. Daniel Povey

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Sheng Zhao
4. Yanqing Liu
5. Shujie Liu
6. Chenpeng Du
7. K. Yu
8. Lei He

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Chenpeng Du
4. Sheng Zhao
5. Dong Yu
6. K. Yu
7. Yanqing Liu
8. Shujie Liu

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “The new codecs include Vocos [62], SpeechTokenizer [63], AudioDec [64], AcademiCodec [65], Descript-audio-codec (DAC) [66], Fun-Codec [67], and RepCodec [68].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “These include Vocos [Siuzdak, 2023], SpeechTokenizer [Zhang et al., 2023a], AudioDec [Wu et al., 2023], AcademiCodec [Yang et al., 2023a], Descript-audio-codec (DAC) [Kumar et al., 2024], FunCodec [Du et al., 2024b], and RepCodec [Huang et al., 2023b].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “For the adversarial loss L adv , we employ both the multi-period discriminator (MPD) and the multi-band multi-scale STFT discriminator, as proposed by [32].”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, neutral): “Concerning the potential misuse of our model, checkpoints will not be released.”
• FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, background): “: descriptinc/descript-audio-codec [20], • AudioDec: facebookresearch/AudioDec [19].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, neutral): “Although all the above models have shown powerful adaptation performance, they have a slow inference speed for their iterative generation manner.”
• Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition (2023, methodology): “As most contemporary neural compression models [14, 15, 16] utilize RVQ quantizer as the basic building block, without loss of generality, we employ EnCodec [15] For baseline, we extract 80-dim mel-spectrogram (Mel-Spec) features using a 512 FFT, a Hann window, 25 ms window length, and 10 ms shift.”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “The acoustic vocoder is separately trained with a design similar to Kumar et al. [2024], Lee et al. [2022], Cong et al. [2021] and Liu and Qian [2021].”
• Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, methodology): “A variant of Encodec, DAC [34], is also adopted for speech resynthesis.”