Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

By Chengyi Wang, Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li and 3 more. IEEE Transactions on Audio, Speech, and Language Processing, 2023

1,289 citations recorded by the source corpus.

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called VALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 50 k hours of English speech which is hundreds of times larger than existing systems. VALL-E emerges in-context learning capability and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as a prompt. Experiment results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find VALL-E could preserve the speaker's emotion and acoustic environment from the prompt in synthesis.

Subfields: Artificial Intelligence, Large Language Models, Speech & Audio

View Paper: https://arxiv.org/abs/2301.02111

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Tao Qin
4. Dong Yu
5. Furu Wei
6. Shinji Watanabe
7. Tie-Yan Liu
8. Wei-Ning Hsu

In Large Language Models

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Wenhui Wang
5. Shuming Ma
6. Tao Qin
7. Xu Tan
8. Xia Song

In Speech & Audio

1. Jinyu Li
2. Wei-Ning Hsu
3. Yossi Adi
4. Xu Tan
5. Sheng Zhao
6. Shinji Watanabe
7. Lei He
8. Rongjie Huang

Citations in the corpus

• Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…systematically categorize these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors +…”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Both representations are compressed (325 bits/s and 400 bits/s respectively) to allow more efficient autoregressive modeling compared to popular audio codecs (e.g. 6k bits/s in [17]).”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “We use language modeling to generate acoustic tokens for the unseen speaker, by learning on the mixed sequence composed of phonemes and codec codes, consistent with previous works (Wang et al., 2023a; Ruben-stein et al., 2023).”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “However, LMs’ ability to process multiple tasks is less exploited in audio generation research: most existing LM-based works are still designed for single tasks (Wang et al., 2023a; Kharitonov et al., 2023).”
• FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, background): “In VALL-E [16], text and speech tokens are joined into a sequence, and a language model is trained to estimate their probabilities.”
• RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “…a result, a growing body of research has begun to incorporate speech tok-enization into the realm of LLM. Noteworthy examples include AudioLM (Borsos et al., 2023), Au-dioPaLM (Rubenstein et al., 2023), Vall-E (Wang et al., 2023a), PolyVoice (Dong et al., 2023), and SpeechGPT (Zhang et al., 2023a).”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “We can see that initializing the model parameters using an exiting VALL-E model checkpoint was beneficial across all tasks, especially in terms of WER.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “It aims to predict speech given context [36, 4] and optionally text guidance [3, 5, 61].”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, background): “FastSpeech 2 has a relatively limited naturalness due to the use of mel-spectrogram, while VALL-E’s performance is limited by the performance of Encodec.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “For the zero-shot experiment, we compared our LibriTTS model to Vall-E [8].”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “We believe this interaction is beneficial when training hierarchical generative models on top of these codes [5, 40, 1], for example to partition the codes into “coarse” tokens (denoting the most significant codes) and “fine” tokens.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “4) Content has monotonic alignment with speech while the autoregressive language model cannot guarantee that, which can lead to repeating or missing word issues [59, 58, 67].”
• PolyVoice: Language Models for Speech to Speech Translation (2023, positive): “Trained on the large corpus of speech data, VALL-E (Wang et al., 2023) leverages the in-context capability of prefix language modeling to achieve state-of-the-art ( sota ) performance for zero-shot speaker cloning.”
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “Moreover, researchers also apply the decoder-only model to text-to-speech synthesis tasks, namely VALL-E (Wang et al., 2023), and VALL-E X (Zhang et al., 2023b).”
• SoundStorm: Efficient Parallel Audio Generation (2023, positive): “…and conditional audio generation as sequence-to-sequence modeling has unlocked rapid progress in speech continuation (Borsos et al., 2022), text-to-speech (Wang et al., 2023; Kharitonov et al., 2023), and general audio and music generation (Kreuk et al., 2022; Agostinelli et al., 2023).”
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, neutral): “Our training data comes from public datasets, such as LibriTTS, VCTK, AISHELL, which mainly includes English and Chinese speech.”