AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios

By Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, Tie-Yan Liu. 2022

78 citations recorded by the source corpus.

Adaptive text to speech (TTS) can synthesize new voices in zero-shot scenarios efficiently, by using a well-trained source TTS model without adapting it on the speech data of new speakers. Considering seen and unseen speakers have diverse characteristics, zero-shot adaptive TTS requires strong generalization ability on speaker characteristics, which brings modeling challenges. In this paper, we develop AdaSpeech 4, a zero-shot adaptive TTS system for high-quality speech synthesis. We model the speaker characteristics systematically to improve the generalization on new speakers. Generally, the modeling of speaker characteristics can be categorized into three steps: extracting speaker representation, taking this speaker representation as condition, and synthesizing speech/mel-spectrogram given this speaker representation. Accordingly, we improve the modeling in three steps: 1) To extract speaker representation with better generalization, we factorize the speaker characteristics into basis vectors and extract speaker representation by weighted combining of these basis vectors through attention. 2) We leverage conditional layer normalization to integrate the extracted speaker representation to TTS model. 3) We propose a novel supervision loss based on the distribution of basis vectors to maintain the corresponding speaker characteristics in generated mel-spectrograms. Without any fine-tuning, AdaSpeech 4 achieves better voice quality and similarity than baselines in multiple datasets.

Subfields: Artificial Intelligence, Natural Language Generation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Zhou Zhao
6. Yi Ren
7. Lei He
8. Jinglin Liu

In Natural Language Generation

1. Frank K. Soong
2. Shujie Liu
3. Yanqing Liu
4. Sheng Zhao
5. Lei He
6. Furu Wei

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yi Ren
4. Tao Qin
5. Zhou Zhao
6. Tie-Yan Liu
7. Lei He
8. Jinglin Liu

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Jinglin Liu
7. Sheng Zhao
8. Lei He

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, background): “Another way is to design advanced but complex speaker encoder [Wu et al., 2022].”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “Cars displayed in these showrooms with ENERGY-EFFICIENT AND GREEN decals prominently featured across the windshield aren’t announcing environmentalism; they’re virtue signaling.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
• NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “Compared to previous methods using reference encoder [44, 45, 46, 11] or pitch/energy extraction [18] for variation information modeling, our posterior encoder φ in VAE acts like a reference encoder that can extract all the necessary variance information in posterior distribution q(z|x;φ).”
• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “In contrast, zero-shot adaptive TTS models utilize in-context learning to generate new voices by exclusively modeling speaker characteristics from a speech prompt (i.e., reference speech) (Wu et al., 2022; Wang et al., 2023; Shen et al., 2023; Li et al., 2023; Le et al., 2023).”