Multi-reference Tacotron by Intercross Training for Style Disentangling, Transfer and Control in Speech Synthesis

By Yanyao Bian, Changbin Chen, Yongguo Kang, Zhenglin Pan. 2019

46 citations recorded by the source corpus.

Speech style control and transfer techniques aim to enrich the diversity and expressiveness of synthesized speech. Existing approaches model all speech styles into one representation, lacking the ability to control a specific speech feature independently. To address this issue, we introduce a novel multi-reference structure to Tacotron and propose intercross training approach, which together ensure that each sub-encoder of the multi-reference encoder independently disentangles and controls a specific style. Experimental results show that our model is able to control and transfer desired speech styles individually.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Frank K. Soong
2. Lei Xie
3. Lei He
4. Shan Yang
5. Xiaochun An
6. Hong-Goo Kang
7. Yinjiao Lei
8. Kyungguen Byun

In Signal Processing

1. Frank K. Soong
2. Lei Xie
3. Xiaochun An
4. Lei He
5. Shan Yang
6. Kyungguen Byun
7. Hong-Goo Kang
8. M. Whitehill

Citations in the corpus

• Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, background): “Thus they can only obtain a global or averaged style [12].”
• Cycle consistent network for end-to-end style transfer TTS training (2021, methodology): “In Bian et al. (2019), multiple references with multiple style ncoders are introduced for style control and transfer.”
• Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency (2019, methodology): “[8], and our random sample is similar to the unpaired training scheme used by Ma et al.”
• Emotional Speech Synthesis with Rich and Granularized Control (2019, background): “, Tacotron), the expressiveness of synthesized speech is controlled by conditioning additional embedding vectors that implicitly provide prosodyrelated latent features typically generated by another neural network [8, 9, 10, 11, 12, 13].”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, methodology): “• MRF-IT: we augment a multi-reference encoder structure into GST-Tacotron 2 [22] and adopt intercross training approach to control and transfer desired speech styles.”
• Controllable Emotion Transfer For End-to-End Speech Synthesis (2020, unknown): “As natural sound can be reasonably produced by current seq2seq-based models learned from a typical corpus with consistently neutral speaking style, there has been increasing interest in how to deliver expressive speech with these seq2seq models [6–10].”