Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias

By Ziyue Jiang, Yi Ren, Zhe Ye, Jinglin Liu, Chen Zhang, Qiang Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin and 2 more. 2023

101 citations recorded by the source corpus.

Scaling text-to-speech to a large and wild dataset has been proven to be highly effective in achieving timbre and speech style generalization, particularly in zero-shot TTS. However, previous works usually encode speech into latent using audio codec and use autoregressive language models or diffusion models to generate it, which ignores the intrinsic nature of speech and may lead to inferior or uncontrollable results. We argue that speech can be decomposed into several attributes (e.g., content, timbre, prosody, and phase) and each of them should be modeled using a module with appropriate inductive biases. From this perspective, we carefully design a novel and large zero-shot TTS system called Mega-TTS, which is trained with large-scale wild data and models different attributes in different ways: 1) Instead of using latent encoded by audio codec as the intermediate feature, we still choose spectrogram as it separates the phase and other attributes very well. Phase can be appropriately constructed by the GAN-based vocoder and does not need to be modeled by the language model. 2) We model the timbre using global vectors since timbre is a global attribute that changes slowly over time. 3) We further use a VQGAN-based acoustic model to generate the spectrogram and a latent code language model to fit the distribution of prosody, since prosody changes quickly over time in a sentence, and language models can capture both local and long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K hours of speech and evaluate its performance on unseen speakers. Experimental results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior naturalness, robustness, and speaker similarity due to the proper inductive bias of each module. Audio samples are available at https://mega-tts.github.io/demo-page.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Sheng Zhao
4. Tao Qin
5. Yi Ren
6. Shujie Liu
7. Tie-Yan Liu
8. Jinglin Liu

In Generative Models

1. Yi Ren
2. Rongjie Huang
3. Jinglin Liu
4. Zhuo Chen
5. Yichong Leng
6. Jinyu Li
7. Jiang Bian
8. Zeqian Ju

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Jinglin Liu
4. Sheng Zhao
5. Rongjie Huang
6. Jinyu Li
7. Yanqing Liu
8. Shujie Liu

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Yi Ren
4. Jinglin Liu
5. Rongjie Huang
6. Sheng Zhao
7. Shujie Liu
8. Zhuo Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Mega-TTS [45] and Mega-TTS 2 [46] proposed the speech disentanglement methods and modeling only partial speech attributes with a language modeling approach.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “Mega-TTS [Jiang et al., 2023b] and Mega-TTS 2 [Jiang et al., 2023a] proposed to first disentangle the multiple attributes in speech, then only model partial attributes with a language modeling approach.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Previous works using raw waveform [8, 9] and mel-spectrogram [1, 2, 10, 7, 11] as data representations suffer from these intricate complexities during speech generation.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “TTS VC SE TSE SVS TT-Sound TT-Music A-Edit SD I-TTS S-Edit YourTTS (Casanova et al., 2022) (Kharitonov et al., 2023) (Huang et al., 2023b) (Jiang et al., 2023) Audio generation is an important component of generative AI.”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Forin-stance,existingspeecheditingmodels[20],[21]arerestrictedto handlingcleansignalsonly,lackingtheabilitytomodifyspoken contentwhilepreservingbackgroundsounds.”
• Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, methodology): “To reduce such a reliance, existing works leverage large-scale generative models trained on datasets at the scale of tens of thousands of hours to perform zero-shot TTS (Kharitonov et al., 2023; Wang et al., 2023a; Shen et al., 2023; Jiang et al., 2023).”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…et al., 2023, Wang et al., 2024a, 2023a] or specialized loss formulations [Ju et al., 2024, Łajszczak et al., 2024] or model designs [Qian et al., 2019, Jiang et al., 2023], our simple self-distillation scheme enables Seed-TTS to achieve high-quality timbre disentanglement without altering model”