Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech

By Chung-Ming Chien, Jheng-hao Lin, Chien-yu Huang, Po-Chun Hsu, Hung-yi Lee. 2021

79 citations recorded by the source corpus.

The few-shot multi-speaker multi-style voice cloning task is to synthesize utterances with voice and speaking style similar to a reference speaker given only a few reference samples. In this work, we investigate different speaker representations and proposed to integrate pretrained and learnable speaker representations. Among different types of embeddings, the embedding pretrained by voice conversion achieves the best performance. The FastSpeech 2 model combined with both pretrained and learnable speaker representations shows great generalization ability on few-shot speakers and achieved 2nd place in the one-shot track of the ICASSP 2021 M2VoC challenge.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Frank K. Soong
6. Lei He
7. Bohan Li
8. Hung-yi Lee

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Frank K. Soong
7. Bohan Li
8. Hung-yi Lee

Citations in the corpus

• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “For the first step, previous zero-shot TTS models [16, 17, 18] usually leverage a speaker/reference encoder to extract speaker representation of a target speaker.”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”
• A Survey on Neural Speech Synthesis (2021, background): “, a lot of works in statistic parametric speech synthesis have studied voice adaptation [77, 386, 443, 65, 122], and the recent voice cloning challenge also attracts a lot of participants [388, 118, 331, 45].”
• Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “[11, 13] studied the effects of different speaker embeddings such as d-vector [16], x-vector [17], LDE-based speaker encoding [13].”
• The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “In T14 [37], the target speaker encoder in AdaIN-VC [38] was adopted to extract speaker information from the target utterance.”