Neural Speech Synthesis with Transformer Network

By Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao. 2018

894 citations recorded by the source corpus.

Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-theart performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves training efficiency. Meanwhile, any two inputs at different times are connected directly by a self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Taming Transformers for High-Resolution Image Synthesis (2020, background): “Originally applied to language tasks, inputs to the transformer were given by tokens, but other signals, such as those obtained from audio [37] or images [8], can be used.”
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “The cascaded TTS systems typically employ a pipeline that involves an acoustic model and a vocoder, using mel spectrograms as intermediate representations [1], [2], [3].”
Semi-Supervised Neural Architecture Search (2020, methodology): “Evaluation metric It has been shown that the quality of the attention alignment between the encoder and decoder is an important influence factor on the quality of synthesized speech in previous works [21, 31, 25, 13, 19], and misalignment can be observed for most mistakes (e.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “Along with the progress of deep learning, significant improvements have been made in TTS research in recent years [Shen et al., 2018, Li et al., 2019, Ren et al., 2019].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Speech Representations in TTS. Traditional works propose using prior-based speech representation such as raw waveform [26, 27, 28] or mel-spectrogram [29, 30, 3, 31].”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In the garden, where the flowers that the gardener who retired last year still bloomed, the children who play there every afternoon find peace and joy.”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “For speech synthesis tasks, AED methods such as Tacotron (Wang et al., 2017), Tacotron2 (Shen et al., 2018), and Trans-formerTTS (Li et al., 2019), and duration-based methods, including Fastspeech (Ren et al., 2019), Fastspeech2 (Ren et al., 2020), and RobuTrans (Li et al., 2020) are widely used.”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Text-to-speech systems [2, 3, 4, 5, 6, 8, 9, 10, 11] aim to generate natural voice with both high quality and diversity.”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Recent years have witnessed signiﬁcant advancements in end-to-end text-to-speech (TTS) synthesis, and the quality of synthesized speech is even close to human parity [Li et al., 2019, Ren et al., 2019, Tan et al., 2022].”
DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, background): “Although twostage TTS systems [2, 3, 4] have been showing good quality in terms of prosody and audio fidelity in the past years, they still suffer several issues: 1) Mel-spectrograms are extracted by Fourier transformation where phase information are lost, which are not an optimal representation to ”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Early TTS models are autoregressive generative models that generate high-quality speech but suffer from a slow synthesis speed due to the sequential sampling procedure (Shen et al., 2018; Li et al., 2019).”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “Following seq2seq TTS models (Li et al., 2019), we enforce the corresponding predicted output Y f , which is generated through the speech-decoder pre-net, Transformer decoder, and speech-decoder post-net, to be close to the original X f by minimizing their L 1 -distance as where x fn denotes n -th…”
A Survey on Neural Speech Synthesis (2021, methodology): “Tacotron 1/2 [72, 73] Ph→MelS NAR / Hybrid/Self-Att/CNN MelNet [361] Ch→MelS AR / RNN DeepVoice [8] Ch/Ph→MelS AR / CNN DeepVoice 2 [84] Ch/Ph→MelS AR / CNN DeepVoice 3 [264] Ch/Ph→MelS AR Seq2Seq CNN ParaNet [262] Ph→MelS NAR Seq2Seq CNN DCTTS [326] Ch→MelS AR Seq2Seq CNN SpeedySpeech [355] Ph→MelS”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “Neural network-based autoregressive TTS systems have shown the capability of synthesizing realistic speech (Shen et al., 2018; Li et al., 2019), but their sequential generative process makes it difﬁcult to fully utilize modern parallel processors.”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “Among feature generators, Tacotron2 (Shen et al., 2018) and Transformer-TTS (Li et al., 2019) enabled highly nat-ural speech synthesis.”
Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “Recently, with unified acoustic and duration modeling, sequenceto-sequence (seq2seq) based neural speech synthesis has achieved superior performance with extraordinary naturalness compared to the conventional methods [4, 5, 6, 7].”
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, methodology): “apid progress. Most neural speech synthesis models use a two-stage pipeline: 1) predicting a low resolution intermediate representation such as mel-spectrograms (Shen et al., 2018, Ping et al., 2017, Li et al., 2019) or linguistic features (Oord et al., 2016) from text, and 2) synthesizing raw wavef”
It's Raw! Audio Generation with State-Space Models (2022, background): “(Right) Pooling layers are position-wise linear transformations with shifts to ensure causality. architecture [39]—remains a central component of state-of-the-art approaches for text-to-speech (TTS) [26], unconditional generation [25], and non-autoregressive (non-AR) generation [23].”