StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

By Cong Han, Vinay S. Raghavan, Gavin Mischler, Nima Mesgarani. 2023

287 citations recorded by the source corpus.

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Lei He
4. Tao Qin
5. Yanqing Liu
6. Tie-Yan Liu
7. Jinyu Li
8. Yichong Leng

In Generative Models

1. Yichong Leng
2. Edresson Casanova
3. Jinyu Li
4. Jiang Bian
5. Zeqian Ju
6. Dongchao Yang
7. Eren Gölge
8. Yanqing Liu

In Speech & Audio

1. Lei He
2. Xu Tan
3. Sheng Zhao
4. Yanqing Liu
5. Jinyu Li
6. Dongchao Yang
7. Yichong Leng
8. Jiang Bian

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “StyleTTS 2 [53], UniCATS [54], NaturalSpeech 2 [55] and NaturalSpeech 3 [56] employed diffusion model [57] for the prompt-conditioned text to speech synthesis.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive [5, 11, 24, 25].”
• XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, methodology): “Wecarried out three training experiments: • Experiment 1: YourTTS model trained only on English us-ing LibriTTS train-clean-460 subset (the same data used in [13]) with the bug on SCL fixed 7 .”
• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “In contrast, zero-shot adaptive TTS models utilize in-context learning to generate new voices by exclusively modeling speaker characteristics from a speech prompt (i.e., reference speech) (Wu et al., 2022; Wang et al., 2023; Shen et al., 2023; Li et al., 2023; Le et al., 2023).”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “The details are described in the following subsections.”
• Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters (2026, unknown): “…reference audio encodings [1]–[3], explicit pitch and duration targets [28], [29], adversarial style discriminators built on SSL representations [4], large-scale speech-token language modeling [5], prosody-focused self-supervised objectives [30], [31], or discrete latent style spaces [32], [33].”