Close to Human Quality TTS with Transformer

By Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu, M. Zhou. 2018

107 citations recorded by the source corpus.

Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-the-art performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves the training efficiency. Meanwhile, any two inputs at different times are connected directly by self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Lei He
8. Yu Zhang

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Lei He
7. Sheng Zhao
8. Yi Ren

Citations in the corpus

• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “With a sufficient amount of studioquality recorded speech data from a single professional speaker, one can train a generative neural end-to-end TTS model capable of synthesizing speech in a reading style almost as natural as the training speaker [4,5].”
• The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “Recent research shows that training ASR and TTS jointly with reconstruction losses can result in improvement in both [15].”
• Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “Recently, Trans-former (Vaswani et al., 2017) has achieved great success and outperformed RNN or CNN based models in many NLP tasks such as neural machine translation and language understanding (Vaswani et al., 2017; Devlin et al., 2018; Li et al., 2018).”
• Multi-reference Tacotron by Intercross Training for Style Disentangling, Transfer and Control in Speech Synthesis (2019, background): “End-to-end-TTS (E2E) models, which generate speech without handcraft linguistic features, are one of the most attractive TTS techniques and have achieved great progresses in voice quality [1, 2, 3].”
• In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data (2019, unknown): “Li et. al.”
• Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, methodology): “For Transformer TTS, we did not use the encoder Prenet in [8] since it did not provide a significant difference in our preliminary experiments.”
• DurIAN: Duration Informed Attention Network for Speech Synthesis (2020, background): “End-to-end approaches [6, 7, 8, 9, 10, 11] have gained much attention recently due to the naturalness of their synthesized results and simplified training pipelines.”
• Forward–Backward Decoding Sequence for Regularizing End-to-End TTS (2019, background): “Various attention mechanisms [32]–[34] have been incorporated to find a more accurate mapping from encoder to decoder.”
• Attention Forcing for Sequence-to-sequence Model Training (2019, methodology): “θ contains a text processing frontend, a duration model and a feature model (Li et al., 2018).”
• Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation (2020, methodology): “Inspired by the structure in paper [3], the structure of PPG predictor is shown in the Fig.”
• On the localness modeling for the self-attention based end-to-end speech synthesis (2020, methodology): “1 illustrates the architecture of the proposed seq2seq framework that shares the basic encoder–decoder architecture with (Li et al., 2019; Vaswani et al., 2017; Yang, Lu et al., 2019).”
• Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion (2020, methodology): “Based on these data, we then explore three architectures – Tacotron2 [1], Transformer [2] and FastSpeech [3] for bilingual and code-switched TTS.”
• Flow-TTS: A Non-Autoregressive Network for Text to Speech Based on Flow (2020, background): “No distillation Tacotron [5] ParaNet [15] Flow-TTS Tacotron 2 [6] FastSpeech [14] Deep Voice 3 [7] TransformerTTS [16] WaveNet [9] ParallelWaveNet [17] WaveGlow [10] WaveRNN [18] ClariNet [8] FloWaveNet [19] LPCNet [20] Table 1 .”
• Light-TTS: Lightweight Multi-Speaker Multi-Lingual Text-to-Speech (2021, methodology): “…fastest speech generation, which extracts phoneme duration sequences from the attention alignment matrixes of a pre-trained autoregressive Transformer TTS [4] model to train a duration predictor, so there is no need for FastSpeech to learn text-to-speech alignments through the attention mechanisms.”
• Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet (2019, background): “The end-to-end TTS system sometimes also uses phone-embedding vectors as the input instead of letter inputs [2, 20].”
• Controlling Emotion Strength with Relative Attribute for End-to-End Speech Synthesis (2019, unknown): “cn acoustics [7,8], texts [9], or both [10] and sample from it with a specified condition, e.”
• A New End-to-End Long-Time Speech Synthesis System Based on Tacotron2 (2019, background): “Speech synthesis has two main goals: intelligibility and naturalness ([12]).”
• Maximizing Mutual Information for Tacotron (2019, background): “Transformer-TTS [5] can be considered as another instance of CAR model, with effective utilization of self-attention mechanism [6].”
• Investigation of Enhanced Tacotron Text-to-speech Synthesis Systems with Self-attention for Pitch Accent Language (2018, methodology): “For the experiment, we built several TTS systems as listed in Table 1.”