A New GAN-based End-to-End TTS Training Algorithm

By Haohan Guo, Frank K. Soong, Lei He, Lei Xie. 2019

49 citations recorded by the source corpus.

End-to-end, autoregressive model-based TTS has shown significant performance improvements over the conventional one. However, the autoregressive module training is affected by the exposure bias, or the mismatch between the different distributions of real and predicted data. While real data is available in training, but in testing, only predicted data is available to feed the autoregressive module. By introducing both real and generated data sequences in training, we can alleviate the effects of the exposure bias. We propose to use Generative Adversarial Network (GAN) along with the key idea of Professor Forcing in training. A discriminator in GAN is jointly trained to equalize the difference between real and predicted data. In AB subjective listening test, the results show that the new approach is preferred over the standard transfer learning with a CMOS improvement of 0.1. Sentence level intelligibility tests show significant improvement in a pathological test set. The GAN-trained new model is also more stable than the baseline to produce better alignments for the Tacotron output.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yu Zhang
6. Ye Jia
7. Heiga Zen
8. Dong Yu

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yu Zhang
6. Ye Jia
7. Heiga Zen
8. Dong Yu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “04 GAN Exposure [96] AM ph AR −→melS AR −→wav IS19 2019.”
• End-to-End Adversarial Text-to-Speech (2020, neutral): “[23] suggest using an adversarial loss to reduce exposure bias [6, 49] in such models.”
• Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “In the past few years, Generative Adversarial Networks (GANs) have shown promising results in image generation [16], [17], [18], speech synthesis [6], [19], [20] and music genera-”
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, background): “(1) The use of the Soft Attention can introduce robustness errors, such as over-generation ( e.g. , word repetitions) and under-generation ( e.g. , word skipping) [5, 11–13].”
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, background): “However, these types of models are known to be less robust than traditional approaches (He et al., 2019; Zheng et al., 2019; Guo et al., 2019; Battenberg et al., 2020).”
• Machine Learning for Synthetic Data Generation: a Review (2023, methodology): “…ImageNet GIRAFFE [35] vision GAN CNN CompCars, LSUN Churches, and FFHQ Wavegrad [36] TTS diffusion gradient-based sampling LJ Speech TTS-GAN [37] TTS GAN auto-regressive model Tacotron2 Seq-GAN [38] NLP GAN+RL CNN Nottingham dataset BLEURT [39] NLP Language model BERT WebNLG Competition…”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, methodology): “A reconstruction loss is used for maintaining reconstruction fidelity of the utterance in the decoder T as: Lrec(θEr , θEz , θT ) = Exs∼Xs [− log pT (xs|zs, rs)] + Ext∼Xt [− log pT (xt|z ∗, rt)] (1) Following GAN [26, 27, 28], we introduce an adversarial loss to be minimized in decoding and adopt a ”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[66] proposed to use the Professor Forcing [116] method for training, which is a GAN-based adversarial training method.”
• Maximizing Mutual Information for Tacotron (2019, background): “In [23], the authors point out that the decoder prenet dropout in Tacotron could make the model condition more on the input text.”
• Teacher-Student Training For Robust Tacotron-Based TTS (2019, methodology): “The techniques to improve out-of-domain performance include the GAN-based TTS framework [25] that introduces both real and generated data sequences in discriminator training, and more recently, stepwise mono-tonic attention for neural TTS [9].”
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech (2020, methodology): “We use the self-attention discriminator proposed in [21] and the hinge version of the adversarial loss [22], both of which have provided good results in image generation [23] and textto-speech [24] tasks, as shown below.”
• Flow-TTS: A Non-Autoregressive Network for Text to Speech Based on Flow (2020, background): “Although teacher forcing techniques [11] are often used in these models to speed up the training process and improve training stability, they cannot speed up the inference process and often lead to mismatches between different distributions of real and predicted data [12, 13].”