Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search

By Jaehyeon Kim, Sungwon Kim, Jungil Kong, Sungroh Yoon. 2020

637 citations recorded by the source corpus.

Recently, text-to-speech (TTS) models such as FastSpeech and ParaNet have been proposed to generate mel-spectrograms from text in parallel. Despite the advantage, the parallel TTS models cannot be trained without guidance from autoregressive TTS models as their external aligners. In this work, we propose Glow-TTS, a flow-based generative model for parallel TTS that does not require any external aligner. By combining the properties of flows and dynamic programming, the proposed model searches for the most probable monotonic alignment between text and the latent representation of speech on its own. We demonstrate that enforcing hard monotonic alignments enables robust TTS, which generalizes to long utterances, and employing generative flows enables fast, diverse, and controllable speech synthesis. Glow-TTS obtains an order-of-magnitude speed-up over the autoregressive model, Tacotron 2, at synthesis with comparable speech quality. We further show that our model can be easily extended to a multi-speaker setting.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Nima Mesgarani
6. Frank K. Soong
7. Sheng Zhao
8. Yi Ren

In Generative Models

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Tao Qin
5. Wei Ping
6. Tie-Yan Liu
7. Jinglin Liu
8. Rongjie Huang

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Sheng Zhao
4. Yi Ren
5. Zhou Zhao
6. Tie-Yan Liu
7. Nima Mesgarani
8. Cong Han

Citations in the corpus

• FedSpeech: Federated Text-to-Speech with Continual Learning (2021, methodology): “From concatenative synthesis with unit selection, statistical parametric synthesis to neural network based parametric synthesis and end-to-end models [ Shen et al. , 2018; Ren et al. , 2019; Ren et al. , 2020; Kim et al. , 2020 ] , the quality of the synthesized speech is closer to the human voice.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Speech Representations in TTS. Traditional works propose using prior-based speech representation such as raw waveform [26, 27, 28] or mel-spectrogram [29, 30, 3, 31].”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Some methods, utilizing techniques such as VAE (Hsu et al., 2019; Lee et al., 2022), flow (Miao et al., 2020; Kim et al., 2020), diffusion (Jeong et al., 2021; Kim et al., 2022; Popov et al., 2021), and others (Wu and Shi, 2022), have achieved promising performance in end-to-end speech synthesis.”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Additional techniques, such as normalizing flow (Valle et al. 2020; Kim et al. 2020) and diffusion models (Liu et al. 2022; Popov et al. 2021; Liu, Guo, and Yu 2023), have been introduced to generate the mel-spectrogram.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “3) YourTTS [17]: This is the zeroshot any-speaker adaptive TTS model based on the flow-based model [2].”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “Then we compare VQTTS with other current popular methods, including Tacotron 2, GlowTTS, FastSpeech 2 and the fully end-to-end TTS model VITS [38].”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, positive): “, 2021), we use a transformer-based text encoder (Kim et al., 2020; Casanova et al., 2021).”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “Owing to the development of non-autoregressive generative models, recent TTS models can generate high-quality speech with faster inference speed (Ren et al., 2019; 2021; Kim et al., 2020; Popov et al., 2021).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Replacing Attention with Duration FastSpeech [284], DurIAN [411], EATS [67] Glow-TTS [156], [190, 19, 355, 422, 73, 298]”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “The duration could be used to train a deterministic duration predictor, as proposed in previous work (Kim et al., 2020), but it cannot express the way a person utters at different speaking rates each time.”
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “Glow-TTS feature generator (Kim et al., 2020) based on Normalizing Flows can be considered as one of the most successful attempts to overcome pronunciation and computational latency issues typical for autoregressive solutions.”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “We build AdaSpeech based on the popular non-autoregressive TTS models (Ren et al., 2019; Peng et al., 2020; Kim et al., 2020; Ren et al., 2021) and further design several techniques to address the challenges in custom voice: • Acoustic condition modeling.”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “Compared with previous non-autoregressive acoustic models [7, 11, 28], most of them focus on improving the duration accuracy to reduce the information gap between the input and output, while FastSpeech 2 and 2s take more missing information into account.”
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech (2021, methodology): “Note that Diff-TTS is about 53% smaller than Tacotron2 and Glow-TTS.”
• NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “We optimize the duration prediction, learnable upsampling layer, and mean/variance linear layers together with the TTS model in a fully differentiable way, which can reduce the training-inference mismatch in previous duration prediction (ground-truth duration is used in training while predicted dura”
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (2021, methodology): “23 MOS gains compared with FastSpeech 2 [25] and Glow-TTS [10] respectively).”