FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec

By Zhihao Du, Shiliang Zhang, Kai Hu, Siqi Zheng. 2023

115 citations recorded by the source corpus.

This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts for the latest neural speech codec models, such as SoundStream and Encodec. Thanks to the unified design with FunASR, FunCodec can be easily integrated into downstream tasks, such as speech recognition. Along with FunCodec, pretrained models are also provided, which can be used for academic or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can achieve comparable speech quality with much lower computation and parameter complexity. Experimental results show that, under the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models. We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/FunCodec.

Subfields: Artificial Intelligence, Speech & Audio, ML Systems & Hardware, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Shujie Liu
5. Yanqing Liu
6. Tao Qin
7. Tie-Yan Liu
8. Furu Wei

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Furu Wei
8. Lei He

In ML Systems & Hardware

1. Jinyu Li

In Signal Processing

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Sanyuan Chen
7. Lei He
8. Furu Wei

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “The new codecs include Vocos [62], SpeechTokenizer [63], AudioDec [64], AcademiCodec [65], Descript-audio-codec (DAC) [66], Fun-Codec [67], and RepCodec [68].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “These include Vocos [Siuzdak, 2023], SpeechTokenizer [Zhang et al., 2023a], AudioDec [Wu et al., 2023], AcademiCodec [Yang et al., 2023a], Descript-audio-codec (DAC) [Kumar et al., 2024], FunCodec [Du et al., 2024b], and RepCodec [Huang et al., 2023b].”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, unknown): “Other details are similar to Du et al. (2023).”
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “Codec language models utilize a speech codec model to extract discrete speech representation [9–11] and employ an autoregressive [8, 12–17] or masked [18] language model to predict the speech tokens, which are then synthesized to waveforms via codec vocoders [19,20].”