GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis

By Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, Zhou Zhao. 2022

36 citations recorded by the source corpus.

Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity. The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting. Audio samples are available at https://GenerSpeech.github.io/

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Tan
2. Dong Yu
3. Chao Weng
4. Ziyue Jiang
5. Yi Ren
6. Qiang Yang
7. Jinyu Li
8. Yongqiang Wang

Citations in the corpus

• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Traditional TTS systems [13, 11, 60, 8, 21] are usually trained on limited datasets, which impairs their models’ ability to produce diverse and generalizable results.”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Adapting models (Chen et al., 2020b; Huang et al., 2022) to a specific individual or object is a long-standing goal in machine learning research.”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “Despite the success achieved, the rising demand for expressive voice generation poses challenges for models, particularly in zero-shot scenarios [5, 14].”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “Besides the study of neural machine translation, many works bring NAR model into other sequence-to-sequence tasks (Cui et al., 2021; Ye et al., 2023; Huang et al., 2022c; Yang et al., 2022b), such as video caption (Yang et al., 2019), speech recognition (Chen et al., 2020) and speech synthesis (Ye…”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “With the fine-grained style features, as well as global speaker embedding, NoreSpeech [127] generates Mel-spectrogram with the same decoder in [27].”
• Zero-Shot Text-to-Speech Synthesis Conditioned Using Self-Supervised Speech Representation Model (2023, methodology): “Although GenerSpeech [21] uses an SSL model as the global”
• Nonparallel Expressive TTS for Unseen Target Speaker using Style-Controlled Adaptive Layer and Optimized Pitch Embedding (2023, background): “Huang et al. [22] introduced a multi-level style adaptor that can transfer speaking style.”