Better speech synthesis through scaling

By James Betker. 2023

128 citations recorded by the source corpus.

In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise -- an expressive, multi-voice text-to-speech system. All model code and trained weights have been open-sourced at https://github.com/neonbjb/tortoise-tts.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Discretization of speech through audio codecs [34– 36] is central to our design, as it enables the direct application of methods developed for LLMs, which underlie recent works on LTTS [16–21 , 37].”
Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, background): “As the speaker is provided by SSVC, with this approach the speech can be generated by conditioning the LLM only with text, and no reference embedding or speech codes are needed like in TorToiSe and VALL-E.”
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, methodology): “Encoder: The GPT-2 encoder is a decoder-only trans-former that is composed of 443M parameters, similar to [12].”
StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “…ﬂow-based modeling [1], [8], [9], explicit modeling of pitch, duration and energy [10], [11], [12], autoregressive language modeling (LM) objectives [13], [14], [15], and using external prosody encoder [16], [17], [18], the production of synthesized speech still falls short of real human speech.”