Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior

By Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg, Bhuvana Ramabhadran, Yonghui Wu. 2020

96 citations recorded by the source corpus.

Recent neural text-to-speech (TTS) models with fine-grained latent features enable precise control of the prosody of synthesized speech. Such models typically incorporate a fine-grained variational autoencoder (VAE) structure, extracting latent features at each input token (e.g., phonemes). However, generating samples with the standard VAE prior often results in unnatural and discontinuous speech, with dramatic prosodic variation between tokens. This paper proposes a sequential prior in a discrete latent space which can generate more naturally sounding samples. This is accomplished by discretizing the latent features using vector quantization (VQ), and separately training an autoregressive (AR) prior model over the result. We evaluate the approach using listening tests, objective metrics of automatic speech recognition (ASR) performance, and measurements of prosody attributes. Experimental results show that the proposed model significantly improves the naturalness in random sample generation. Furthermore, initial experiments demonstrate that randomly sampling from the proposed model can be used as data augmentation to improve the ASR performance.

Subfields: Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

1. Yi Ren
2. Chenpeng Du
3. Zhou Zhao
4. Rongjie Huang
5. Jinglin Liu
6. K. Yu
7. Xu Tan
8. Tao Qin

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Tao Qin
4. Tie-Yan Liu
5. Lei He
6. Yi Ren
7. Zhou Zhao
8. Chenpeng Du

Citations in the corpus

• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, neutral): “[19] suggest modeling style using an autoregressive prior over representations from quantized fine-grained VAE and perform evaluation by synthesizing utterances for training ASR on LibriSpeech.”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “The effectiveness of VQVAE is also explored in the related literature [17, 18].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “They fine-tune the whole model (Chen et al., 2018; Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters.”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Modeling these variation information [17, 3, 18, 4] can mitigate the one-to-many mapping problem and thus improve expressiveness and fidelity of synthesized speech.”
• Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, background): “It has also been shown that prosodically diverse synthesized speech effectively improves ASR performance [12].”
• Mixture Density Network for Phone-Level Prosody Modelling in Speech Synthesis (2021, methodology): “In order to provide better voice quality in the synthetic speech, we scale the predicted standard deviations of the Gaussians with a factor of 0.2 when sampling, following [12].”
• Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis (2021, methodology): “One basic method of PLP modelling is to use a single Gaussian [19], [20].”
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, background): “The notable results are shown not only for single speaker TTS modeling [3, 4, 5, 6, 7], multi-speaker TTS modeling [8, 9] but also for cloning prosody style [10, 11, 12, 13, 14] and speaker characteristics [15, 16, 17, 18].”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Sun et al. [42, 41] adopt VAE to represent the fine-grained prosody variable, naturally enabling sampling of different prosody features for each phoneme.”
• Rich Prosody Diversity Modelling with Phone-Level Mixture Density Network (2021, methodology): “One basic method of PLP modelling is using a single Gaussian [10, 12].”
• Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis (2020, methodology): “b © Neural-based prosody features: a reference encoder is used to drain information from the ground-truth utterance [1, 2, 3, 4, 5, 9].”
• Unsupervised Word-Level Prosody Tagging for Controllable Speech Synthesis (2022, methodology): “[14] uses vector quantization and trains an autoregressive prior model to generate synthetic speech with better sound quality.”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, background): “[34], [35] study a way to include a hierarchical, fine-grained prosody representation, that represents the recent attempts to”
• Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS (2021, background): “Recent advancement of end-to-end neural TTS has demonstrated that it can synthesize very natural, human-like speech [1, 2, 3, 4, 5].”
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “The previous approaches [15, 16, 17, 18, 19, 22, 23] aimed to control the speech characteristics of the output speech through an encoder.”
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, methodology): “A recent study [17] used an AR prior network conditioned on the phone embeddings, which is trained to fit the VAE posterior distribution.”