Teacher-Student Training For Robust Tacotron-Based TTS

By Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, Haizhou Li. 2019

40 citations recorded by the source corpus.

While neural end-to-end text-to-speech (TTS) is superior to conventional statistical methods in many ways, the exposure bias problem in the autoregressive models remains an issue to be resolved. The exposure bias problem arises from the mismatch between the training and inference process, that results in unpredictable performance for out-of-domain test data at run-time. To overcome this, we propose a teacher-student training scheme for Tacotron-based TTS by introducing a distillation loss function in addition to the feature loss function. We first train a Tacotron2-based TTS model by always providing natural speech frames to the decoder, that serves as a teacher model. We then train another Tacotron2-based model as a student model, of which the decoder takes the predicted speech frames as input, similar to how the decoder works during run-time inference. With the distillation loss, the student model learns the output probabilities from the teacher model, that is called knowledge distillation. Experiments show that our proposed training scheme consistently improves the voice quality for out-of-domain test data both in Chinese and English systems.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Lei He
7. Yanqing Liu
8. Junichi Yamagishi

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Lei He
6. Sheng Zhao
7. Yanqing Liu
8. Haizhou Li

Citations in the corpus

• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, positive): “Instead of feeding all ground-truth features during training, we leverage a schedule sampling mechanism [19] for pitch, utterance-level acous-”
• A Survey on Neural Speech Synthesis (2021, methodology): “[204] conduct teacher-student distillation [108, 160, 337] to reduce the exposure bias problem, where the teacher is trained with teacher-forcing mode, and the student takes the previously predicted value as input and is optimized to reduce the distance of hidden states between the teacher and stude”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[127] also proposed a method based on knowledge distillation to reduce exposure bias, which is to train a teacher model first, and then use it to guide the training of the student model.”
• Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022, background): “…et al., 2016; Shen et al., 2016; Bahdanau et al., 2017; Chen et al., 2020), learning to search (Leblond et al., 2018), energy-based models (Deng et al., 2020), imitation learning (Du and Ji, 2019), generative adversarial networks (Yu et al., 2017) and knowledge distillation (Liu et al., 2019).”
• Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS (2020, background): “Prosody, if taken into consideration, is modeled from the statistics of the training data [3], [4], [6], [8].”
• Graphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis (2020, background): “Tacotron-based neural TTS [8, 9] and its variants [10–13] are such examples.”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “Tacotron [89], [180], [218]–[220] represents one of the successful text-to-speech (TTS) implementations, that has been extended to voice conversion [3], [183], [221].”