BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

By Mateusz Lajszczak, Guillermo Cámbara, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín-Cortinas, Ammar Abbas and 10 more. 2024

125 citations recorded by the source corpus.

We introduce a text-to-speech (TTS) model called BASE TTS, which stands for $\textbf{B}$ig $\textbf{A}$daptive $\textbf{S}$treamable TTS with $\textbf{E}$mergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported"emergent abilities"of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated by the model can be heard at https://amazon-ltts-paper.com/.

Subfields: Artificial Intelligence, Large Language Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Lei He
4. Shujie Liu
5. Sheng Zhao
6. Tao Qin
7. Zhuo Chen
8. Yanqing Liu

In Large Language Models

1. Furu Wei
2. Jinyu Li
3. Sanyuan Chen
4. Shujie Liu
5. Long Zhou
6. Huaming Wang
7. Sheng Zhao
8. Xu Tan

In Speech & Audio

1. Lei He
2. Xu Tan
3. Sheng Zhao
4. Yanqing Liu
5. Jinyu Li
6. Shujie Liu
7. Zhuo Chen
8. Sanyuan Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “UniAudio [49] and BASE TTS [50] further explored scaling the codec language model to 1b parameters and 100 k hours of training data.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “UniAudio [Yang et al., 2023b] and BASE TTS [Łajszczak et al., 2024] further explored scaling the codec language model to 1b parameters and 100k hours of training data.”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…applied methods such as feature engineering [Chen et al., 2023, Wang et al., 2024a, 2023a] or specialized loss formulations [Ju et al., 2024, Łajszczak et al., 2024] or model designs [Qian et al., 2019, Jiang et al., 2023], our simple self-distillation scheme enables Seed-TTS to achieve…”
• Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters (2026, unknown): “…reference audio encodings [1]–[3], explicit pitch and duration targets [28], [29], adversarial style discriminators built on SSL representations [4], large-scale speech-token language modeling [5], prosody-focused self-supervised objectives [30], [31], or discrete latent style spaces [32], [33].”