eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer
By Ammar Abbas, Sri Karlapati, Bastian Schnell, Penny Karanasou, M. G. Moya, A. Nagaraj, Ayman Boustati, Nicole Peinelt, Alexis Moinet, Thomas Drugman. 2023
3 citations recorded by the source corpus.
We present eCat, a novel end-to-end multispeaker model capable of: a) generating long-context speech with expressive and contextually appropriate prosody, and b) performing fine-grained prosody transfer between any pair of seen speakers. eCat is trained using a two-stage training approach. In Stage I, the model learns speaker-independent word-level prosody representations in an end-to-end fashion from speech. In Stage II, we learn to predict the prosody representations using the contextual information available in text. We compare eCat to CopyCat2, a model capable of both fine-grained prosody transfer (FPT) and multi-speaker TTS. We show that eCat statistically significantly reduces the gap in naturalness between CopyCat2 and human recordings by an average of 46.7% across 2 languages, 3 locales, and 7 speakers, along with better target-speaker similarity in FPT. We also compare eCat to VITS, and show a statistically significant preference.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Mateusz Lajszczak
- Sri Karlapati
- M. G. Moya
- Ammar Abbas
- Guillermo Cámbara
- Panagiota Karanasou
- Guillermo Cámbara
- Alexis Moinet
Citations in the corpus
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “In training, we use the same set of adversarial and non-adversarial losses as [23].”
- A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “For the prosody predictor, we used the training hyperparameters from the original eCat [21], which were tuned for RoBERTa-base.”