Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis

By Rafael Valle, Kevin J. Shih, Ryan J. Prenger, Bryan Catanzaro. 2020

135 citations recorded by the source corpus.

In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer. Flowtron borrows insights from IAF and revamps Tacotron in order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns an invertible mapping of data to a latent space that can be manipulated to control many aspects of speech synthesis (pitch, tone, speech rate, cadence, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of-the-art TTS models in terms of speech quality. In addition, we provide results on control of speech variation, interpolation between samples and style transfer between speakers seen and unseen during training. Code and pre-trained models will be made publicly available at this https URL

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Zhou Zhao
7. Yi Ren
8. Wei-Ning Hsu

In Generative Models

1. Xu Tan
2. Wei Ping
3. Tao Qin
4. Tie-Yan Liu
5. Frank K. Soong
6. Bryan Catanzaro
7. Sungroh Yoon
8. Wei-Ning Hsu

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Yi Ren
7. Adrian La'ncucki
8. Zhou Zhao

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Wei Ping
6. Sheng Zhao
7. Wei-Ning Hsu
8. Lei He

Citations in the corpus

• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Additional techniques, such as normalizing flow (Valle et al. 2020; Kim et al. 2020) and diffusion models (Liu et al. 2022; Popov et al. 2021; Liu, Guo, and Yu 2023), have been introduced to generate the mel-spectrogram.”
• A Survey on Neural Speech Synthesis (2021, methodology): “05 Flowtron [360] AM ph AR −→melS FF −→wav ICLR21 2020.”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “Autoregressive TTS models (Taigman et al., 2018; Shen et al., 2018; Valle et al., 2021) generate diverse speech with different rhythms through their autoregressive structure and several tricks including maintaining dropout probability during inference and priming (Graves, 2013).”
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, methodology): “For this reason, most current TTS models are designed for the English language (Kim et al., 2020; Ping et al., 2017; Shen et al., 2018; Valle et al., 2020), which is a language with many open resources.”
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, methodology): “The main difference from our work is that Flowtron uses autoregressive flows as its objective is not about fast speech synthesis.”
• End-to-End Adversarial Text-to-Speech (2020, background): “Tacotron 1 & 2 [54, 65], Deep Voice 2 & 3 [17, 44], TransformerTTS [34] and Flowtron [59] are autoregressive models that generate spectrograms frame by frame.”
• EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture (2020, methodology): “The most popular neural TTS models are autoregressive models based on an encoder-decoder framework (Wang et al., 2017; Shen et al., 2018; Ping et al., 2018; 2019; Li et al., 2019; Valle et al., 2020).”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Valle R, Shih K, Prenger R, Catanzaro B (2020) Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:200505957 220.”
• Generative Pre-training for Speech with Flow Matching (2023, methodology): “Besides neural vocoders, generative models are also applied to other tasks such as TTS (Valle et al., 2020), speech enhancement (Lu et al., 2021; 2022; Richter et al., 2023) and separation Scheibler et al. (2023).”
• One TTS Alignment to Rule Them All (2021, methodology): “For autoregressive models like Flowtron and Tacotron 2, we compare with the baseline alignment methods therein.”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “Many approaches were recently proposed to improve modelling of expressive speech [15, 16, 17, 18].”
• Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis (2020, methodology): “Recent work has integrated normalizing flows with sequence-tosequence TTS models, to enable a non-autoregressive decoder [15], or improve modelling of the intermediate mel spectrogram [16, 17].”
• ItôWave: Itô Stochastic Differential Equation is all You Need for Wave Generation (2022, background): “It was found that although the score network model does not have as strict restrictions on the network structure as the flow model [16, 7, 17], not all network structures are suitable for score prediction.”
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “Several approaches have been proposed to address such a problem, including methods based on variational inference [1, 4, 5, 6], flow-based modeling [1, 7, 8], controlling pitch, duration and energy [9, 10], and using external prosody encoder [11, 12, 13].”
• RAD-TTS: Parallel Flow-Based TTS with Robust Alignment Learning and Diverse Synthesis (2021, methodology): “It can be constructed using either another bi-partite flow for full parallelism, or an autoregressive flow similar to the architecture in (Valle et al., 2020).”