CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer

By Sri Karlapati, Penny Karanasou, Mateusz Lajszczak, Ammar Abbas, Alexis Moinet, Peter Makarov, Raymond Li, Arent van Korlaar, Simon Slangen, Thomas Drugman. 2022

17 citations recorded by the source corpus.

In this paper, we present CopyCat2 (CC2), a novel model capable of: a) synthesizing speech with different speaker identities, b) generating speech with expressive and contextually appropriate prosody, and c) transferring prosody at fine-grained level between any pair of seen speakers. We do this by activating distinct parts of the network for different tasks. We train our model using a novel approach to two-stage training. In Stage I, the model learns speaker-independent word-level prosody representations from speech which it uses for many-to-many fine-grained prosody transfer. In Stage II, we learn to predict these prosody representations using the contextual information available in text, thereby, enabling multi-speaker TTS with contextually appropriate prosody. We compare CC2 to two strong baselines, one in TTS with contextually appropriate prosody, and one in fine-grained prosody transfer. CC2 reduces the gap in naturalness between our baseline and copy-synthesised speech by $22.79\%$. In fine-grained prosody transfer evaluations, it obtains a relative improvement of $33.15\%$ in target speaker similarity.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Mateusz Lajszczak
2. Sri Karlapati
3. M. G. Moya
4. Arnaud Joly
5. Guillermo Cámbara
6. Ammar Abbas
7. Panagiota Karanasou
8. Penny Karanasou

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “CNN journalists tracking the stories behind the officials who served during the tumultuous period when the protests rocked the nation to its core noticed significant inconsistencies in the official reports provided.”
• A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, background): “[20] present CopyCat2, a TTS system that learns a prosodic space and subsequently predicts prosody representations based on contextualized subword embeddings from BERT.”
• eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer (2023, background): “In the realm of prosody representation learning, there has been significant work in learning speaker independent prosody representations for many-to-many fine-grained prosody transfer (FPT) [17–19].”
• Controllable Emphasis with zero data for text-to-speech (2023, background): “While context awareness of TTS system has vastly improved (see [3], [4] among others), automated output does not always assign the correct intonation to cases like (1e), given preceding context .”