Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
By Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Z. Chen, Yu Zhang, Yuxuan Wang, R. Skerry-Ryan and 3 more. 2017
3,120 citations recorded by the source corpus.
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text. The system is composed of a recurrent sequence-to-sequence feature prediction network that maps character embeddings to mel-scale spectrograms, followed by a modified WaveNet model acting as a vocoder to synthesize time-domain waveforms from those spectrograms. Our model achieves a mean opinion score (MOS) of 4.53 comparable to a MOS of 4.58 for professionally recorded speech. To validate our design choices, we present ablation studies of key components of our system and evaluate the impact of using mel spectrograms as the conditioning input to WaveNet instead of linguistic, duration, and $F_{0}$ features. We further show that using this compact acoustic intermediate representation allows for a significant reduction in the size of the WaveNet architecture.
Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Kyunghyun Cho
- Jianfeng Gao
- Tao Qin
- Tie-Yan Liu
- Quoc V. Le
- Luke S. Zettlemoyer
In Speech & Audio
Citations in the corpus
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “For years, the fields have been continuously reporting new state of the art results using varieties of model architectures for computer vision tasks [57, 58, 7], for natural language understanding tasks [59, 60, 61], for speech recognition and synthesis tasks [62, 63, 64, 65, 66].”
- Jukebox: A Generative Model for Music (2020, methodology): “More recent works like Deep Voice 3 (Ping et al., 2018), Tacotron 2 (Shen et al., 2018), and Char2Wav (Sotelo et al., 2017) learn speech synthesis end-to-end using sequence-to-sequence architecture (Sutskever et al., 2014).”
- Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, background): “The first step transforms the text into time-aligned features, such as a mel-spectrogram [4, 5], or F0 frequencies and other linguistic features [2, 6].”
- DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “In fact, previous study (Shen et al., 2018) suggests that even a moderate large receptive field size (e.”
- WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “We chose a female speaker in the training dataset, who had also been used in (Oord et al., 2016; Kalchbrenner et al., 2018; Shen et al., 2018; Bikowski et al., 2020), for evaluation.”
- You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “We chose Tacotron [24], [7] as a base for our speech synthesizer.”
- Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, methodology): “We retrieve the embedding of each phoneme in Ypair from the codebook to obtain the ground truth phoneme embedding sequence Qpair and trained the decoder with standard sequence-to-sequence TTS objective [17].”
- Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
- Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, methodology): “The architecture is the same as that described in [15], composed of 30 dilated convolution layers.”
- Neural Speech Synthesis with Transformer Network (2018, background): “Those MOS tests are rigorous and reliable, as each audio is listened to by at least 20 testers, who are all native English speakers (comparing to Tacotron2’s 8 testers in Shen et al. (2017)), and each tester listens less than 30 audios.”
- Semi-Supervised Neural Architecture Search (2020, methodology): “Evaluation metric It has been shown that the quality of the attention alignment between the encoder and decoder is an important influence factor on the quality of synthesized speech in previous works [21, 31, 25, 13, 19], and misalignment can be observed for most mistakes (e.”
- FedSpeech: Federated Text-to-Speech with Continual Learning (2021, methodology): “Aiming to synthesize natural speech, text to speech (TTS) [Shen et al. , 2018; Yamamoto et al. , 2020] remains a crucial research topic.”
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “Along with the progress of deep learning, significant improvements have been made in TTS research in recent years [Shen et al., 2018, Li et al., 2019, Ren et al., 2019].”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Traditional TTS systems [1, 2, 3, 4] are typically trained on limited datasets recorded in studios, and thus fail to support high-quality zero-shot speech synthesis.”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “The chef seasoning the fish tossed it gently.”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “However, these models encode style in a low-dimensional space and impose an overly simple distribution of speech given text and residual embedding [51, 56].”
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “The typical neural text-to-speech (TTS) pipeline, such as Tacotron 2 (Shen et al. 2018) and FastSpeech 2 (Ren et al. 2020), consists of two stages: predicting the mel-spectrogram from text and then vocoding it into waveform.”