Interactive Text-to-Speech via Semi-supervised Style Transfer Learning

By Yang Gao, Weiyi Zheng, Zhaojun Yang, Thilo Köhler, Christian Fuegen, Qing He. 2020

11 citations recorded by the source corpus.

With increasing interests in interactive speech systems, speech emotion recognition and multi-style text-to-speech (TTS) synthesis are becoming increasingly important research areas. In this paper, we combine both. We present a method to extract speech style embed-dings from input speech queries and apply this embedding as conditional input to a TTS voice so that the TTS response matches the speaking style of the input query. Specifically, we first train a multi-modal style classification model using acoustic and textual features of speech utterances. Due to a limited amount of labeled data, we combined the emotional recognition dataset: the interactive emotional dyadic motion capture database (IEMOCAP) with a small labeled subset of our internal TTS dataset for style model training. We take the softmax layer from the style classifier as style embedding and then apply this style embedding extraction model to generate soft style labels for our unlabelled internal TTS dataset. With this semi-supervised approach, reliable style embeddings are extracted to train a multi-style TTS system. As a result, we developed a controllable multi-style TTS system whose response matches the given target styles embedding, which could be extracted from the input query or manually assigned.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Rui Liu
6. Sheng Zhao
7. Haizhou Li
8. Berrak Sisman

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Rui Liu
6. Berrak Sisman
7. Haizhou Li
8. Guanglai Gao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Prosody/style/emotion is the key information to improve the expressiveness of speech and the vast majority of works on expressive TTS focus on improving the prosody/style/emotion of speech [303, 377, 315, 82, 353, 318].”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, methodology): “[80] propose to use an emotion recognizer to extract the style embedding for speech style transfer.”