From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint

By Zexin Cai, Chuxiong Zhang, Ming Li. 2020

44 citations recorded by the source corpus.

High-fidelity speech can be synthesized by end-to-end text-to-speech models in recent years. However, accessing and controlling speech attributes such as speaker identity, prosody, and emotion in a text-to-speech system remains a challenge. This paper presents a system involving feedback constraint for multispeaker speech synthesis. We manage to enhance the knowledge transfer from the speaker verification to the speech synthesis by engaging the speaker verification network. The constraint is taken by an added loss related to the speaker identity, which is centralized to improve the speaker similarity between the synthesized speech and its natural reference audio. The model is trained and evaluated on publicly available datasets. Experimental results, including visualization on speaker embedding space, show significant improvement in terms of speaker identity cloning in the spectrogram level. Synthesized samples are available online for listening. (this https URL)

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Hung-yi Lee
6. Sheng Zhao
7. Da-Rong Liu
8. Yi-Chen Chen

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Hung-yi Lee
6. Yi-Chen Chen
7. Sung-Feng Huang
8. Da-Rong Liu

Citations in the corpus

• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Technique Description Work Disentangling with Adversarial Training Disentanglement for control [218, 117, 275, 427] Cycle Consistency/Feedback for Control Enhance style/timbre generation [198, 380, 202, 30, 191] Semi-Supervised Learning for Control Use VAE and adversarial training [100, 116, 117, 42”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, unknown): “[77] study to use a pre-trained speaker”
• Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “[18] measured the speaker similarity between the predicted mel-spectrogram and the reference.”