FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
By Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu. 2020
1,809 citations recorded by the source corpus.
Advanced text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs during training and use predicted values during inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of full end-to-end training and even faster inference than FastSpeech. Experimental results show that 1) FastSpeech 2 and 2s outperform FastSpeech in voice quality with much simplified training pipeline and reduced training time; 2) FastSpeech 2 and 2s can match the voice quality of autoregressive models while enjoying much faster inference speed.
Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Model Architectures
In Speech & Audio
In Signal Processing
Citations in the corpus
- FedSpeech: Federated Text-to-Speech with Continual Learning (2021, methodology): “Although neural network-based TTS models [Shen et al., 2018; Ren et al., 2020] can generate high-quality speech samples, they suffer from the lack of robustness (e.”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “The movie that De Moya who was recently awarded the lifetime achievement award starred in 2022 was a box-office hit, despite the mixed reviews.”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “For tasks like TTS, SE, TSE, TT-Music, VC, S-Edit, SD, SVS: (1) their neural network architectures are based on transformer (Ren et al., 2020) or others (Oord et al., 2016; Luo & Mesgarani, 2019); (2) their training objectives can be either in time-domain (Luo & Mesgarani, 2019), frequency-domain (Y”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “On the other hand, previous studies have shown that regression duration models can produce reasonable speech [51, 37].”
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “This process follows the idea introduced in FastSpeech 2 (Ren et al. 2020).”
- High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “To mitigate this problem, audio generation is typically divided into two stages: 1) predicting audio conditioned on some intermediate representation such as mel-spectrograms [24, 28, 19, 30] and 2) predicting the intermediate representation given some conditioning information, such as text [35, 34].”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “…models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30, 37].”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “For speech synthesis tasks, AED methods such as Tacotron (Wang et al., 2017), Tacotron2 (Shen et al., 2018), and Trans-formerTTS (Li et al., 2019), and duration-based methods, including Fastspeech (Ren et al., 2019), Fastspeech2 (Ren et al., 2020), and RobuTrans (Li et al., 2020) are widely used.”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “We calculate the prosody similarity with the following steps: 1) we extract phoneme-level duration and pitch from the prompt and the synthesized speech; 2) we calculate the mean, standard variation, skewness, and kurtosis [7] of the pitch and duration in each speech sequence; 3) we calculate the dif”
- Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “Recently, the deep neural network-based Text-To-Speech (TTS) synthesis models have shown remarkable performance on both quality and speed, thanks to the progress on the generative modeling [1, 2], non-autoregressive acoustic models [3], and the powerful neural vocoder [4].”
- VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “Two well-known AMs are Tacotron 2 [4] based on encoderattention-decoder architecture and FastSpeech 2 [5] based on Transformer blocks.”
- AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “Neural text to speech (TTS) [1] models can synthesize high quality human voice when being trained with a large amount of single-speaker or multi-speaker datasets [2, 3, 4, 5, 6, 7, 8, 9].”
- A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “…from one of this speaker’s other audio exam-2 ples. systems become capable of generating (Oord et al., 2016; Shen et al., et ; Peng et al., 2019; Ren et al., 2020). generally consist of two stages: stage which generates an intermediate acoustic representation (linear-or mel-spectrogram)…”
- YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, neutral): “For efficient end-to-end vocoder training, we randomly sample constant length partial sequences from z as in [23, 27, 28, 19].”
- Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Owing to the development of non-autoregressive generative models, recent TTS models can generate high-quality speech with faster inference speed (Ren et al., 2019; 2021; Kim et al., 2020; Popov et al., 2021).”
- Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “We use FastSpeech 2 [3] as our TTS model architecture, which is one of the most popular single-speaker non-autoregressive TTS models.”
- A Survey on Neural Speech Synthesis (2021, methodology): “, Tacotron 1/2 [376, 297], Deep Voice 3 [264], and FastSpeech 1/2 [284, 286]) are proposed to simplify text analysis modules and directly take character/phoneme sequences as input, and simplify acoustic features with mel-spectrograms.”
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “In practice, we do not upsam-ple the whole latent variables z but use partial sequences as an input for the decoder, which is the windowed generator training used for efficient end-to-end training (Ren et al., 2021; Donahue et al., 2021).”