RepCodec: A Speech Representation Codec for Speech Tokenization

By Zhichao Huang, Chutong Meng, Tom Ko. 2023

50 citations recorded by the source corpus.

With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore, this superiority extends across various speech encoders and languages, affirming the robustness of RepCodec. We believe our method can facilitate large language modeling research on speech processing.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Shujie Liu
5. Yanqing Liu
6. Tao Qin
7. Tie-Yan Liu
8. Furu Wei

In Representation & Self-Supervised Learning

1. Jinyu Li
2. Sanyuan Chen
3. Shujie Liu
4. Long Zhou
5. Thomas Drugman
6. Xu Tan
7. Yao Qian
8. Furu Wei

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Furu Wei
8. Lei He

In Signal Processing

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Lei He
8. Furu Wei

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “The new codecs include Vocos [62], SpeechTokenizer [63], AudioDec [64], AcademiCodec [65], Descript-audio-codec (DAC) [66], Fun-Codec [67], and RepCodec [68].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “These include Vocos [Siuzdak, 2023], SpeechTokenizer [Zhang et al., 2023a], AudioDec [Wu et al., 2023], AcademiCodec [Yang et al., 2023a], Descript-audio-codec (DAC) [Kumar et al., 2024], FunCodec [Du et al., 2024b], and RepCodec [Huang et al., 2023b].”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “While John, who had been working late every night for a month on his novel, finally took a break to enjoy the fresh air, his neighbor, a painter who often found inspiration in the midnight moon, was just beginning her creative process.”