Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation

By Jianhua Tao, Ruibo Fu, Jiangyan Yi, Zhengqi Wen, Rongxiu Zhong. 2020

24 citations recorded by the source corpus.

The low similarity and naturalness of synthesized speech remain a challenging problem for speaker adaptation with few resources. Since the acoustic model is too complex to interpret, overfitting will occur when training with few data. To prevent the model from overfitting, this paper proposes a novel speaker adaptation framework that decomposes the parameter space of the end-to-end acoustic model into two parts, with the one on predicting spoken content and the other on modeling speaker’s voice. The spoken content is represented by phone posterior-gram (PPG) which is speaker independent. By adapting the two sub-modules separately, the overfitting can be alleviated effectively. Moreover, we propose two different adaptation strategies based on whether the data has text annotation. In this way, speaker adaptation can also be performed without text annotations. Experimental results confirm the adaptability of our proposed method of factorizating spoken content and voice. Lis-tening tests demonstrate that our proposed method can achieve better performance with just 10 sentences than speaker adaptation conducted on Tacotron in terms of naturalness and speaker similarity.

Subfields: Artificial Intelligence, Speech & Audio, Meta & Continual Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Tao Qin
5. Shujie Liu
6. Tie-Yan Liu
7. Yanqing Liu
8. Furu Wei

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Hung-yi Lee
7. Lei He
8. Sung-Feng Huang

In Meta & Continual Learning

1. Hung-yi Lee
2. Xu Tan

In Signal Processing

1. Xu Tan
2. Sheng Zhao
3. Yanqing Liu
4. Jinyu Li
5. Shujie Liu
6. Tao Qin
7. Hung-yi Lee
8. Sanyuan Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Subsequent research in speaker adaptation has focused on improving efficiency, requiring less data and fewer speaker-specific parameters [6], [7], [20].”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “Early work in zero-shot TTS typically employed speaker adaptation and speaker encoding methods, which often required additional fine-tuning, complex pre-designed features, or heavy structure engineering [Chen et al., 2019, Wang et al., 2020, Arik et al., 2018, Casanova et al., 2022].”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”
• The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “In [12], phone posteriorgram (PPG) is used as an intermediate variable to factor-ize spoken content and voice [12].”