Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

By Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo. 2018

145 citations recorded by the source corpus.

Recent advances in neural autoregressive models have improve the performance of speech synthesis (SS). However, as they lack the ability to model global characteristics of speech (such as speaker individualities or speaking styles), particularly when these characteristics have not been labeled, making neural autoregressive SS systems more expressive is still an open issue. In this paper, we propose to combine VoiceLoop, an autoregressive SS model, with Variational Autoencoder (VAE). This approach, unlike traditional autoregressive SS systems, uses VAE to model the global characteristics explicitly, enabling the expressiveness of the synthesized speech to be controlled in an unsupervised manner. Experiments using the VCTK and Blizzard2012 datasets show the VAE helps VoiceLoop to generate higher quality speech and to control the expressions in its synthesized speech by incorporating global characteristics into the speech generating process.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. James R. Glass
7. Wei-Ning Hsu
8. Yu Zhang

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Junichi Yamagishi
4. Xu Tan
5. Lei He
6. Mohit Bansal
7. James R. Glass
8. Heiga Zen

In Generative Models

1. Wei-Ning Hsu
2. Junichi Yamagishi
3. James R. Glass
4. Jonathan Ho
5. Tao Qin
6. Xu Tan
7. Yu Zhang
8. Bryan Catanzaro

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “Ideas like clustering (Dehak et al., 2011), priming (Wang et al., 2018), and variational autoencoders (Hsu et al., 2019; Akuzawa et al., 2018) have been used to learn broader styles of speech and control expressivity in an unsupervised way.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “They cannot generate realistic noisy speech given a low dimensional vector, and performance degrades when conditioned on noisy references [21].”
• A Survey on Neural Speech Synthesis (2021, background): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
• Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, neutral): “The proposed GMVAE-Tacotron is most related to Skerry-Ryan et al. (2018), Wang et al. (2018), Henter et al. (2018), and Akuzawa et al. (2018), which introduce a reference embedding to model prosody or noise.”
• MelNet: A Generative Model for Audio in the Frequency Domain (2019, background): “…frame as a diagonal Gaussian with parameterized mean µ i ∈ R d and standard deviation σ i ∈ R d + , where d is the dimension of each frame: • VAE: Global z Subsequent works [1, 23] variational autoencoder (VAE) [30] to jointly model a latent variable z which conditions the generation of each frame.”
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, background): “Although VAEs and GANs (Hsu et al., 2018; Bińkowski et al., 2019; Akuzawa et al., 2018) based models also provide a latent prior that can be easily manipulated, in Flowtron this comes at no cost in speech quality nor optimization challenges.”
• Taming VAEs (2018, background): “Deep variational auto-encoders (VAEs) have made substantial progress in the last few years on modelling complex data such as images [1, 2, 3, 4, 5, 6, 7], speech synthesis [8, 9], molecular discovery [10, 11, 12], robotics [13, 14, 15] and 3d-scene understanding [16, 17, 18].”
• Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis (2018, methodology): “[12] which combines an autoregressive speech synthesis model with VAE for expressive speech synthesis.”
• Rich Prosody Diversity Modelling with Phone-Level Mixture Density Network (2021, methodology): “1) ULP: A recent popular approach for utterance-level prosody (ULP) modelling is conditioning the synthesis on a latent variable from VAE [8].”
• Predicting Expressive Speaking Style from Text in End-To-End Speech Synthesis (2018, background): “[20] K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “VAELoopDemo: audio samples generated by VAE-Loop,” https://akuzeee.github.io/VAELoopDemo/, 2018.”
• Using Vaes and Normalizing Flows for One-Shot Text-To-Speech Synthesis of Expressive Speech (2019, unknown): “Although there is existing work in the related fields of prosody generation [4] and prosody transfer [5], these approaches rely on either modelling expressive speech or transferring a seen expressive style to an unseen speaker, and are trained on a large corpora of expressive speech.”
• The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “Variational Autoen-coder (VAE) [6, 7] and GST [8] are two typical models built upon sequence-to-sequence (seq2seq) models for style modeling.”
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features (2019, methodology): “[11] conditioned the encoder of VoiceLoop [12] with a latent variable containing global speech characteristics.”
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, methodology): “A VAE is applied to a TTS synthesis system by regarding X and Y as the sequence of acoustic features and the linguistic features, respectively [12, 13, 14].”
• Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis (2020, background): “style or prosody embeddings to achieve a fine-grained control of the synthesized speech [19–24].”
• Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “VAE has successful applications in image generation [50], motion prediction [51], text generation [52], and expressive speech generation [53].”
• Self-supervised Context-aware Style Representation for Expressive Speech Synthesis (2022, unknown): “The unsupervised approach models styles based on joint training with both reference audio and text content [1, 2, 3, 4, 5].”
• Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis (2020, background): “Abundant research has been performed on learning latent representations for styles and prosody [15, 16] such as the use of an utterancelevel VAE in [17].”
• Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization (2019, methodology): “Recent development of neural end-to-end TTS models [28, 2] enables control of both labelled and 17 unlabelled speech attributes by conditioning synthesis on both text and learned attribute represen18 tations [29, 23, 12, 1, 7, 11].”