StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis
By Cong Han, Nima Mesgarani. 2022
81 citations recorded by the source corpus.
Text-to-Speech (TTS) has recently seen great progress in synthesizing high-quality speech owing to the rapid development of parallel TTS systems. Yet producing speech with naturalistic prosodic variations, speaking styles, and emotional tones remains challenging. In addition, many existing parallel TTS models often struggle with identifying optimal monotonic alignments since speech and duration generation typically occur independently. Here, we propose StyleTTS, a style-based generative model for parallel TTS that can synthesize diverse speech with natural prosody from a reference speech utterance. Using our novel Transferable Monotonic Aligner (TMA) and duration-invariant data augmentation, StyleTTS significantly outperforms other baseline models on both single and multi-speaker datasets in subjective tests of speech naturalness and synthesized speaker similarity. It also demonstrates higher robustness and emotional similarity to the reference speech as indicated by word error rate (WER) and acoustic feature correlations. Through self-supervised learning, StyleTTS can generate speech with the same emotional and prosodic tone as the reference speech without needing explicit labels for these categories. In addition, when trained with a large number of speakers, our model can perform zero-shot speaker adaption. The source code and audio samples can be found on our demo page at https://styletts.github.io/.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Cong Han
- Nima Mesgarani
- Gavin Mischler
- Vinay S. Raghavan
- Xilin Jiang
- Yi Luo
- John R. Hershey
- Jonathan Le Roux
Citations in the corpus
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We replace the mel-discriminator in [6] with the multi-period discriminator (MPD) [30] and multi-resolution discriminator (MRD)”
- STYLETTS-VC: ONE-SHOT VOICE CONVERSION BY KNOWLEDGE TRANSFER FROM STYLE-BASED TTS MODELS (2022, methodology): “An overview of StyleTTS-VC is provided in Figure 1b.”
- PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “These MOS scores are significantly worse than those reported in [ 3 ] by roughly 0.4 to 0.5 points.”