YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

By Edresson Casanova, Julian Weber, Christopher Dane Shulby, Arnaldo Cândido Júnior, Eren Gölge, Moacir Antonelli Ponti. 2021

643 citations recorded by the source corpus.

YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. Our method builds upon the VITS model and adds several novel modifications for zero-shot multi-speaker and multilingual training. We achieved state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. Additionally, our approach achieves promising results in a target language with a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS and zero-shot voice conversion systems in low-resource languages. Finally, it is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality. This is important to allow synthesis for speakers with a very different voice or recording characteristics from those seen during training.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Jinyu Li
3. Tao Qin
4. Tie-Yan Liu
5. Dong Yu
6. Wei-Ning Hsu
7. Sheng Zhao
8. Furu Wei

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Sheng Zhao
4. Wei-Ning Hsu
5. Rongjie Huang
6. Yi Ren
7. Zhou Zhao
8. Lei He

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “To tackle the zero-shot TTS problem, existing work leverages speaker adaptation and speaker encoding methods, requiring additional fine-tuning, complex pre-designed features, or heavy structure engineering [6], [7], [8], [9].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Early work in zero-shot TTS typically employed speaker adaptation and speaker encoding methods, which often required additional fine-tuning, complex pre-designed features, or heavy structure engineering [Chen et al., 2019, Wang et al., 2020, Arik et al., 2018, Casanova et al., 2022].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “…adv L adv + λ feat L feat + λ codebook L codebook + λ commit L commit + λ ph L ph + λ f0 L f0 + λ gr-ph L gr-ph + λ gr-f0 L gr-f0 + λ gr-spk L gr-spk , where λ rec , λ adv , λ feat , λ codebook , λ commit , λ f0 , λ ph , λ gr-f0 , λ gr-ph and λ gr-spk are coefficients for balancing each loss terms.”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Cars displayed in these showrooms with ENERGY-EFFICIENT AND GREEN decals prominently featured across the windshield aren’t announcing environmentalism; they’re virtue signaling.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “TTS VC SE TSE SVS TT-Sound TT-Music A-Edit SD I-TTS S-Edit YourTTS (Casanova et al., 2022) (Kharitonov et al., 2023) (Huang et al., 2023b) (Jiang et al., 2023) Audio generation is an important component of generative AI.”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Early studies in zero-shot TTS employed fixed-dimensional speaker embeddings [11], [12], [13], [14].”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “2) YourTTS [Casanova et al., 2021], SOTA multilingual (English, French, and Portuguese) zero-shot TTS model trained on VCTK, LibriTTS, TTS-Portugese [Casanova et al.”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Additionally, we compute the Speaker En-coder Cosine Similarity (SECS) (Casanova et al. 2022) as an auxiliary metric to assess speaker similarity.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Our baseline models consisted of the three highest-performing public models: VITS [3], StyleTTS [6], and JETS [32] for LJSpeech; and VITS, YourTTS [60], and StyleTTS for LibriTTS.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “We compare the zero-shot synthesis performance of Mega-TTS with baseline systems, including: 1)YourTTS [8], a powerful zero-shot TTS model trained on 1k hours of speech dataset.”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “2) Although YourTTS has seen 97 from 108 speakers in VCTK dataset, our model can still outperform it by a large margin, which demonstrates the advantages of NaturalSpeech 2.”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Baselines We adopt YourTTS 12 [Casanova et al., 2022] as our baseline for zero-shot cross-lingual TTS. YourTTS is a zero-shot multi-speaker TTS model for everyone, whose speaker information is based on speaker embedding extracted from a reference speech.”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “For objective evaluation, we use Speaker Embedding Cosine Similarity (SECS) and Character Error Rate (CER) metrics used in YourTTS [17].”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, methodology): “Some previous works [19, 20] apply speaker classification loss on generated mel-spectrograms as a supervision to ensure the synthesized speech to be more similar to reference speech, which, however, does not bring much improvement in speaker similarity [16].”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “…audio; 2) YourTTS (Casanova et al., 2022), Gen-erSpeech (Huang et al., 2022b), VALL-E (Wang et al., 2023a) for English zero-shot TTS; 3) YourTTS (Casanova et al., 2022) for zero-shot multilingual TTS; 4) NANSY (Choi et al., 2022) and PPG-VC (Liu et al., 2021) for VC; 5) Diff-singer (Liu et al.,…”
• Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “Following the methodology of Wang et al. (2023) we compute the cosine similarity between embeddings computed from the prompt (encoded and decoded with SoundStream) and from the generated speech, using a publicly available speaker verification system based on WavLM (Chen et al.,”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “Despite the success achieved, the rising demand for expressive voice generation poses challenges for models, particularly in zero-shot scenarios [5, 14].”
• Generative Pre-training for Speech with Flow Matching (2023, methodology): “In particular, we focus on the zero-shot speaker adaptation problem (Jia et al., 2018; Casanova et al., 2021) where the voice of an unseen speaker should be used for synthesis.”
• LM-VC: Zero-Shot Voice Conversion via Speech Generation Based on Language Models (2023, methodology): “We also include a recent state-of-the-art VC system YourTTS [22] with an open-source checkpoint as another comparison system.”