Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis

By Yinjiao Lei, Shan Yang, Lei Xie. 2020

66 citations recorded by the source corpus.

This paper proposes a unified model to conduct emotion transfer, control and prediction for sequence-to-sequence based fine-grained emotional speech synthesis. Conventional emotional speech synthesis often needs manual labels or reference audio to determine the emotional expressions of synthesized speech. Such coarse labels cannot control the details of speech emotion, often resulting in an averaged emotion expression delivery, and it is also hard to choose suitable reference audio during inference. To conduct fine-grained emotion expression generation, we introduce phoneme-level emotion strength representations through a learned ranking function to describe the local emotion details, and the sentence-level emotion category is adopted to render the global emotions of synthesized speech. With the global render and local descriptors of emotions, we can obtain fine-grained emotion expressions from reference audio via its emotion descriptors (for transfer) or directly from phoneme-level manual labels (for control). As for the emotional speech synthesis with arbitrary text inputs, the proposed model can also predict phoneme-level emotion expressions from texts, which does not require any reference audio or manual label.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Rui Liu
6. Haizhou Li
7. Lei Xie
8. Berrak Sisman

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Haizhou Li
6. Berrak Sisman
7. Sheng Zhao
8. Lei Xie

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Rui Liu
6. Berrak Sisman
7. Haizhou Li
8. Lei Xie

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Information Granularity Language/Speaker Level Multi-lingual/speaker TTS [438, 241, 38] Paragraph Level Long-form reading [11, 389, 370] Utterance Level Timbre/Prosody/Noise [303, 377, 139, 315, 202, 39] Word/Syllable Level Fine-grained information [319, 113, 44, 329] Character/Phoneme Level [318, 4”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “Due to the lack of data, researchers mostly use internal databases for emotional text-to-speech studies (Um et al., 2020; Lei et al., 2021; Kwon et al., 2019; Zhu et al., 2019).”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, background): “, speaker identity, emotion and prosody), which are essential for expressive and diverse voice generation, can be also modelled and controlled [8, 9, 10, 11, 12, 13].”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, background): “Finally, a particular line of work uses text sentiment to control the synthesized emotion [135], [157] or its strength [123].”