Learning Latent Representations for Style Control and Transfer in End-to-end Speech Synthesis

By Ya-Jie Zhang, Shifeng Pan, Lei He, Zhenhua Ling. 2018

249 citations recorded by the source corpus.

In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, background): “In speech synthesis, Hsu et al. (2019) and Zhang et al. (2019) combine Tacotron 2 and VAEs to learn speech style and prosody.”
Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, background): “Since natural-sounding can be reasonably produced by current seq2seq-based speech synthesis models learned from a typical corpus with neutral speaking style, there have been increasing interests in how to deliver expressive speeches with these seq2seq models [8, 9, 10, 11].”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “To stabilize the training process, we adopted KL annealing proposed in [30].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “[112] Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling.”
Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis (2020, background): “b © Neural-based prosody features: a reference encoder is used to drain information from the ground-truth utterance [1, 2, 3, 4, 5, 9].”
Controllable Emotion Transfer For End-to-End Speech Synthesis (2020, unknown): “Following the principle of ‘say it like this’ [6], there has evolved a plenty of work in this direction lately [29], such as Global Style Tokens (GST) [7, 15], Variational Autoencoder (VAE) [8, 30] and their variants and updates [31, 32].”
A Review of Deep Learning Based Speech Synthesis (2019, background): “The authors in [70] introduced deep generative models, such as Variational Auto-encoder (VAE) [71], to Tacotron to explicitly model the latent representation of a speaker state in a continuous space, and additionally to control the speaking style in speech synthesis [70].”
Improving transfer of expressivity for end-to-end multispeaker text-to-speech synthesis (2021, methodology): “For training the end-to-end TTS system, we have used the same model parameters as explained in [3], [5], [14] for implementing the Tacotron 2 system and expressivity encoders based on GST, and VAE.”
Multi-reference Tacotron by Intercross Training for Style Disentangling, Transfer and Control in Speech Synthesis (2019, methodology): “The unsupervised way, which combines E2E models and reference encoders into Autoencoder (AE) [5] or Variational Autoencoder (VAE) [6, 7, 8], gets closer results.”
Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis (2021, background): “The works of [27, 28] pursue similar goals although, being based on globalsentence level embeddings, they do not address fine-level control as we do.”
Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, background): “In previous work (Hsu et al., 2019; Zhang et al., 2019), the variational posterior has the form q ( z x ) , which matches the form of the true posterior for a simple generative model p ( x z ) p ( z ) .”
Fully-Hierarchical Fine-Grained Prosody Modeling For Interpretable Speech Synthesis (2020, background): “Abundant research has been performed on learning latent representations for styles and prosody [15, 16] such as the use of an utterancelevel VAE in [17].”
Emotional Speech Synthesis with Rich and Granularized Control (2019, background): “, Tacotron), the expressiveness of synthesized speech is controlled by conditioning additional embedding vectors that implicitly provide prosodyrelated latent features typically generated by another neural network [8, 9, 10, 11, 12, 13].”
Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, methodology): “A VAE is applied to a TTS synthesis system by regarding X and Y as the sequence of acoustic features and the linguistic features, respectively [12, 13, 14].”
Speech Bert Embedding for Improving Prosody in Neural TTS (2021, background): “Meanwhile, prosody can also be enriched during prosody transfer, like [8, 9, 10, 11].”
STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech (2021, background): “In order to resolve such a problem, several approaches [7, 8, 9, 10, 11] has been explored upon TTS models so that they can synthesize and control speech with various style.”
Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, background): “While some of the approaches [13, 14, 15, 16, 17] rely on prosodic features such as fundamental frequency, duration and energy to generate the expressive speech but these approaches can generate expressive speech for the speakers who are already part of the training.”
Nonparallel Expressive TTS for Unseen Target Speaker using Style-Controlled Adaptive Layer and Optimized Pitch Embedding (2023, background): “To address this issue, various techniques [9] [11] [12] have been explored to synthesize and control speech with diverse styles.”