CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech

By Sri Karlapati, Alexis Moinet, Arnaud Joly, Viacheslav Klimkov, Daniel Sáez-Trigueros, Thomas Drugman. 2020

74 citations recorded by the source corpus.

Prosody Transfer (PT) is a technique that aims to use the prosody from a source audio as a reference while synthesising speech. Fine-grained PT aims at capturing prosodic aspects like rhythm, emphasis, melody, duration, and loudness, from a source audio at a very granular level and transferring them when synthesising speech in a different target speaker's voice. Current approaches for fine-grained PT suffer from source speaker leakage, where the synthesised speech has the voice identity of the source speaker as opposed to the target speaker. In order to mitigate this issue, they compromise on the quality of PT. In this paper, we propose CopyCat, a novel, many-to-many PT system that is robust to source speaker leakage, without using parallel data. We achieve this through a novel reference encoder architecture capable of capturing temporal prosodic representations which are robust to source speaker leakage. We compare CopyCat against a state-of-the-art fine-grained PT model through various subjective evaluations, where we show a relative improvement of $47\%$ in the quality of prosody transfer and $14\%$ in preserving the target speaker identity, while still maintaining the same naturalness.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Chenpeng Du
5. Frank K. Soong
6. K. Yu
7. Sheng Zhao
8. Thomas Drugman

In Experimental and Cognitive Psychology

1. Tao Qin
2. Chenpeng Du
3. Frank K. Soong
4. Tie-Yan Liu
5. Thomas Drugman
6. K. Yu
7. Xu Tan

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Chenpeng Du
5. K. Yu
6. Frank K. Soong
7. Jaime Lorenzo-Trueba
8. Raymond Li

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “The movie that De Moya who was recently awarded the lifetime achievement award starred in 2022 was a box-office hit, despite the mixed reviews.”
• A Survey on Neural Speech Synthesis (2021, methodology): “In this subsection, we review techniques on disentangling [218, 117, 275], controlling [353, 180, 240, 13, 267, 343, 192], and transferring [150, 131, 393, 6] variation information, as shown in Table 15.”
• Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis (2021, methodology): “To alleviate the speaker identity interference, [23] introduces an instance norm layer in its prosody extractor and a bottleneck layer that downsamples the prosody representations with a factor τ .”
• eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer (2023, background): “Karla-pati et al. [18] used a conditional variational autoencoder to learn speaker-independent prosody representations at the frame-level to capture source prosody.”
• CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer (2022, methodology): “We found that the average number of mel-spectrogram frames a word spans over was approximately equal to the fixed downsampling rate, β, of CC1.”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “The authors in [8, 9] applied the CopyCat VC model [14] to generate expressive synthetic speech in low-resource data settings.”
• Low-Resource Expressive Text-To-Speech Using Data Augmentation (2020, methodology): “To create synthetic data for our experiments, we leverage the Copycat model [25] as VC model (Fig.”