NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

By Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang and 9 more. 2024

403 citations recorded by the source corpus.

While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech in a zero-shot way. Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt. With this factorization design, NaturalSpeech 3 can effectively and efficiently model intricate speech with disentangled subspaces in a divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, and achieves on-par quality with human recordings. Furthermore, we achieve better performance by scaling to 1B parameters and 200K hours of training data.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Shujie Liu
4. Sheng Zhao
5. Zhuo Chen
6. Yanqing Liu
7. Sanyuan Chen
8. Tao Qin

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Yanqing Liu
5. Shujie Liu
6. Zhuo Chen
7. Dongchao Yang
8. Sanyuan Chen

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Zhuo Chen
4. Shujie Liu
5. Sheng Zhao
6. Sanyuan Chen
7. Takuya Yoshioka
8. Yanqing Liu

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “StyleTTS 2 [53], UniCATS [54], NaturalSpeech 2 [55] and NaturalSpeech 3 [56] employed diffusion model [57] for the prompt-conditioned text to speech synthesis.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “To improve modeling efficiency, some works explore fully non-autoregressive methods for zero-shot TTS [Borsos et al., 2023, Le et al., 2024, Ju et al., 2024].”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “…work by expanding the tasks supported, enhancing robustness and quality (e.g., stabilizing the AR generation with duration control [41] and leveraging improved neural codecs [42]), and developing more advanced conditioning mechanisms (e.g., us-ing disentangled prosody and speaker information [43]).”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “This is quite different from TTS methods based on non-autoregressive (NAR) generative models [12, 16, 25], which predicts all tokens at the same time, thus have high robustness but relatively low diversity.”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “…We compare our models with leading TTS systems including, (mainly) autoregressive models: VALL-E 2 [13], MELLE [29], FireRedTTS [63] and CosyVoice [64]; non-autoregressive models: Voicebox [14], NaturalSpeech 3 [12], DiTTo-TTS [35], MaskGCT [65], Seed-TTS DiT [39] and our reproduced E2 TTS [36].”