Hierarchical Generative Modeling for Controllable Speech Synthesis

By Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Z. Chen, Jonathan Shen and 2 more. 2018

295 citations recorded by the source corpus.

This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Jinyu Li
5. Yu Zhang
6. Dong Yu
7. Frank K. Soong
8. Shinji Watanabe

In Generative Models

1. Wei Ping
2. Xu Tan
3. Wei-Ning Hsu
4. Jonathan Ho
5. Tao Qin
6. Yi Ren
7. Tie-Yan Liu
8. Hung-yi Lee

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Xu Tan
4. Tao Qin
5. Yu Zhang
6. Tie-Yan Liu
7. Frank K. Soong
8. Junichi Yamagishi

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “…MidiNet (Yang et al., 2017) and MuseGAN (Dong et al., 2018) which use generative adversarial networks, MusicVAE (Roberts et al., 2018) and HRNN (Wu et al., 2019) which use hierarchical recurrent networks, and Music Transformer (Huang et al., 2019a) and MuseNet (Payne, 2019) which use…”
• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “Using GMVAE-Tacotron as a speech synthesizer and modified LPCNet as a vocoder, we generated 360 hours of synthetic speech with random prosody, modelled by a variational autoencoder.”
• Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “1In TensorFlow this can be conveniently implemented using zq(x) = ze(x) + stop_gradient(eq(x) − ze(x)) autoregressive decoder [33], [46] or the encoder can periodically emit vectors of latent features to be consumed by the decoder”
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “Some methods, utilizing techniques such as VAE (Hsu et al., 2019; Lee et al., 2022), flow (Miao et al., 2020; Kim et al., 2020), diffusion (Jeong et al., 2021; Kim et al., 2022; Popov et al., 2021), and others (Wu and Shi, 2022), have achieved promising performance in end-to-end speech synthesis.”
• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “This feature is often desired as exemplified in earlier work where emotion and voice can be controlled independently (Hsu et al., 2019; Kulkarni et al., 2021; Nguyen et al., 2023).”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “They cannot generate realistic noisy speech given a low dimensional vector, and performance degrades when conditioned on noisy references [21].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “In speech synthesis, Hsu et al. (2019) and Zhang et al. (2019) combine Tacotron 2 and VAEs to learn speech style and prosody.”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, methodology): “Thus, the focus ofTTS research is shifting towardmore challenging tasks, such asmulti-speaker TTS [6–8], building TTS systems from small amounts of data [9], few shot voice adaptation [8, 10, 11], unsupervised prosody and speaking style modeling [12, 13], and the use of found data [13, 14].”
• MelNet: A Generative Model for Audio in the Frequency Domain (2019, background): “…frame as a diagonal Gaussian with parameterized mean µ i ∈ R d and standard deviation σ i ∈ R d + , where d is the dimension of each frame: • VAE: Global z Subsequent works [1, 23] variational autoencoder (VAE) [30] to jointly model a latent variable z which conditions the generation of each frame.”
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, methodology): “A variation of this approach is proposed by (Hsu et al., 2018), wherein a Gaussian mixture model (GMM) encoding the audio is added to Tacotron to learn a latent embedding.”
• Generating Diverse and Natural Text-to-Speech Samples Using a Quantized Fine-Grained VAE and Autoregressive Prosody Prior (2020, methodology): “0% GMVAE-Tacotron [8] copy synthesis 16.”
• Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “Two different VAE models are investigated; a global VAE per speaker similar to [28] and a fine-grained phoneme-level VAE akin to [30].”
• MultiSpeech: Multi-Speaker Text to Speech with Transformer (2020, methodology): “To reduce deployment and serving cost in commercial applications, building a TTS system supporting multiple (hundreds or thousands) speakers has attracted much attention in both industry and academia [6, 7, 3, 8].”
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation (2021, methodology): “Moreover, GMVAE-Tacotron (Hsu et al., 2019) present a variational approach with Gaussian Mixture prior in style modeling.”
• Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, background): “Incorporating recent advances in TTS to introduce more natural prosody and style variation [31, 32, 33] to the synthetic speech might further improve performance when training on synthetic speech.”
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, background): “The notable results are shown not only for single speaker TTS modeling [3, 4, 5, 6, 7], multi-speaker TTS modeling [8, 9] but also for cloning prosody style [10, 11, 12, 13, 14] and speaker characteristics [15, 16, 17, 18].”