Emotional Speech Synthesis with Rich and Granularized Control

By Seyun Um, Sangshin Oh, Kyungguen Byun, Inseon Jang, Chunghyun Ahn, Hong-Goo Kang. 2019

97 citations recorded by the source corpus.

This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Rui Liu

In Experimental and Cognitive Psychology

1. Panagiotis Tzirakis
2. Rui Liu
3. Haizhou Li
4. Berrak Sisman
5. Kun Zhou
6. Tao Qin
7. Lei Xie
8. Björn Schuller

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Rui Liu
6. Berrak Sisman
7. Haizhou Li
8. Zhaoxi Mu

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “In these experiments, we adopt the source prompts from the emotional voices dataset EmoV-DB [Um et al., 2020] as inputs of VALL-E X Trans to generate the translated target speech, whose samples are listed on our demo page.”
• A Survey on Neural Speech Synthesis (2021, methodology): “In this subsection, we review techniques on disentangling [218, 117, 275], controlling [353, 180, 240, 13, 267, 343, 192], and transferring [150, 131, 393, 6] variation information, as shown in Table 15.”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, methodology): “…that an average style is computed on all training data and stored as a reference for each category, therefore sparing the need to select an appropriate reference during inference though some works attempt to derive more representative styles than simply averaging the context vectors [155], [156].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[214] proposed to improve the method of simply averaging the style embedding vectors belonging to each emotion category [115] to determine the representative weight vectors by maximizing the ratio of inter-category distance to intra-category distance (I2I), and proposed to apply the spread-aware I2I”
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, background): “cn/opencpop/ 16], emotional speech [17, 18, 19, 20, 21, 22], and also singing voice [23, 24].”
• Controllable Emotion Transfer For End-to-End Speech Synthesis (2020, unknown): “Following the principle of ‘say it like this’ [6], there has evolved a plenty of work in this direction lately [29], such as Global Style Tokens (GST) [7, 15], Variational Autoencoder (VAE) [8, 30] and their variants and updates [31, 32].”
• Improving Emotional TTS with an Emotion Intensity Input from Unsupervised Extraction (2021, methodology): “The resulting attention matrix is aggregated with five 2D convolutional layers with [30, 30, 30, 10, 6] output channels, a 5 × 5 kernel size, and a stride of 2 × 2 .”
• Text aware Emotional Text-to-speech with BERT (2022, methodology): “[6] proposes using the centroid of GST embed-dings for each emotion and [7] shows that the GST embeddings for each emotion could be highly dispersed and proposes using an inter-to-intra distance ratio algorithm that considers the distribution of each emotion.”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “Due to the lack of data, researchers mostly use internal databases for emotional text-to-speech studies (Um et al., 2020; Lei et al., 2021; Kwon et al., 2019; Zhu et al., 2019).”