Controllable Emotion Transfer For End-to-End Speech Synthesis

By Tao Li, Shan Yang, Liumeng Xue, Lei Xie. 2020

91 citations recorded by the source corpus.

Emotion embedding space learned from references is a straight-forward approach for emotion transfer in encoder-decoder structured emotional text to speech (TTS) systems. However, the transferred emotion in the synthetic speech is not accurate and expressive enough with emotion category confusions. Moreover, it is hard to select an appropriate reference to deliver desired emotion strength. To solve these problems, we propose a novel approach based on Tacotron. First, we plug two emotion classifiers – one after the reference encoder, one after the decoder output – to enhance the emotion-discriminative ability of the emotion embedding and the predicted mel-spectrum. Second, we adopt style loss to measure the difference between the generated and reference mel-spectrum. The emotion strength in the synthetic speech can be controlled by adjusting the value of the emotion embedding as the emotion embedding can be viewed as the feature map of the mel-spectrum. Experiments on emotion transfer and strength control have shown that the synthetic speech of the proposed method is more accurate and expressive with less emotion category confusions and the control of emotion strength is more salient to listeners.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Hung-yi Lee

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Hung-yi Lee
6. Rui Liu
7. Zhaoxi Mu
8. Xinyu Yang

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “[191] conduct controllable emotional transfer by adding an emotion style classifier with feedback cycle, where the classifier encourages the TTS model to synthesize speech with specific emotion.”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, methodology): “In Liu et al. (2021), an emotional text-to-speech framework with reinforcement learning is built and evaluated through the comparative study with several state-of-the-art emotional text-to-speech frameworks (Li et al., 2021; Cai et al., 2021) on the ESD database.”
• Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech (2021, methodology): “Some previous works suggest using multi-task training, such as jointly optimizing the TTS framework and an auxiliary classification task [12, 13].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, background): “In addition to combining TTS and ASR [123, 173, 208, 209, 232], it is also possible to combine speaker recognition with multi-speaker TTS [30, 209], and combine speech emotion recognition with emotional speech synthesis [120] for dual training.”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, background): “On the unseen style transfer of TR2C and TR2P, the GST, VAE and MRF-IT, MRF-ACC models are not capable of keeping the speaker’s timbre, resulting in low similarity scores.”