PortaSpeech: Portable and High-Quality Generative Text-to-Speech

By Yi Ren, Jinglin Liu, Zhou Zhao. 2021

96 citations recorded by the source corpus.

Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel. After analyzing two kinds of generative NAR-TTS models (VAE and normalizing flow), we find that: VAE is good at capturing the long-range semantics features (e.g., prosody) even with small model size but suffers from blurry and unnatural results; and normalizing flow is good at reconstructing the frequency bin-wise details but performs poorly when the number of model parameters is limited. Inspired by these observations, to generate diverse speech with natural details and rich prosody using a lightweight architecture, we propose PortaSpeech, a portable and high-quality generative text-to-speech model. Specifically, 1) to model both the prosody and mel-spectrogram details accurately, we adopt a lightweight VAE with an enhanced prior followed by a flow-based post-net with strong conditional inputs as the main architecture. 2) To further compress the model size and memory footprint, we introduce the grouped parameter sharing mechanism to the affine coupling layers in the post-net. 3) To improve the expressiveness of synthesized speech and reduce the dependency on accurate fine-grained alignment between text and speech, we propose a linguistic encoder with mixture alignment combining hard inter-word alignment and soft intra-word alignment, which explicitly extracts word-level semantic information. Experimental results show that PortaSpeech outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M (about 4x model size and 3x runtime memory compression ratio compared with FastSpeech 2). Our extensive ablation studies demonstrate that each design in PortaSpeech is effective.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rongjie Huang
2. Zhou Zhao
3. Yi Ren
4. Jinglin Liu
5. Dong Yu
6. Chenye Cui
7. Songxiang Liu
8. Dan Su

Citations in the corpus

• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Generative models, like language models [4, 33], VAE [34, 47], GAN [31, 30], Normalizing flow [39, 29], and diffusion model [32, 24, 43, 22], have been applied to speech or audio synthesis for years.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, background): “, the limited sample quality [37] or sample diversity [35].”
• M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, unknown): “Recently, benefited from the rapid development of deep learning and the continuous breakthroughs in TTS techniques [47; 23; 52; 37; 35; 36; 18], singing voice synthesis [45; 21; 14; 5; 38; 2; 27; 40; 9; 13; 10; 51; 17; 8; 43; 1; 22; 48; 54; 41; 19; 56; 53; 25] and its related tasks (e.g., SVC [33;…”
• Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, background): “Thanks to the advance of non-autoregressive models [4, 5] and powerful generative models [6, 7, 8, 9], the audio quality and inference speed of modern TTS has been greatly improved.”
• DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, background): “In several studies [1], [2], [3], [4], [5], [6], [7], prosody modeling has been leveraged to synthesize speech that closely resembles human expression.”
• DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs (2022, methodology): “Other generative models have also been studied for TTS, such as Flow-based models(Kim et al., 2020; Miao et al., 2020), variational auto-encoder (VAE)-based models(Lee et al., 2021; Liu et al., 2021b), and generative adversarial network (GAN)-based models (Donahue et al., 2021; Yang et al., 2021).”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, unknown): “3) to reconstruct details in these expressive speech samples, we include a flow-based post-net (Ren et al., 2021) to refine the transform decoder output and generate fine-grained mel-spectrograms.”