Towards Multi-Scale Style Control for Expressive Speech Synthesis

By Xiang Li, Jingbei Li, Zhiyong Wu, Jia Jia, Helen Mei-Ling Meng. 2021

52 citations recorded by the source corpus.

This paper introduces a multi-scale speech style modeling method for end-to-end expressive speech synthesis. The proposed method employs a multi-scale reference encoder to extract both the global-scale utterance-level and the local-scale quasi-phoneme-level style features of the target speech, which are then fed into the speech synthesis model as an extension to the input phoneme sequence. During training time, the multi-scale style model could be jointly trained with the speech synthesis model in an end-to-end fashion. By applying the proposed method to style transfer task, experimental results indicate that the controllability of the multi-scale speech style model and the expressiveness of the synthesized speech are greatly improved. Moreover, by assigning different reference speeches to extraction of style on each scale, the flexibility of the proposed method is further revealed.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Rongjie Huang
7. Jinglin Liu
8. Frank K. Soong

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “In this subsection, we review techniques on disentangling [218, 117, 275], controlling [353, 180, 240, 13, 267, 343, 192], and transferring [150, 131, 393, 6] variation information, as shown in Table 15.”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Inspired by [6], many similar works propose to learn speaking style from a reference audio, such as, Li et al. [30] use a multi-scale style encoder capturing style information from reference audio to facilitate expressive speech synthesis; Huang et al. [29] propose a multi-level style adaptor to…”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “These TTS models have been extended to more complex scenarios, including multiple speakers, emotions, and styles for expressive and diverse voice generation (Chen et al., 2020a; Li et al., 2021).”
• Nonparallel Expressive TTS for Unseen Target Speaker using Style-Controlled Adaptive Layer and Optimized Pitch Embedding (2023, methodology): “Wang et al. [9] suggested using global style tokens to manage and transfer the overall style, while Li et al. [18] incorporated a multi-scale style encoder to improve expressive speech synthesis.”