Predicting Expressive Speaking Style from Text in End-To-End Speech Synthesis

By Daisy Stanton, Yuxuan Wang, R. Skerry-Ryan. 2018

129 citations recorded by the source corpus.

Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the Text-Predicting Global Style Token (TP-GST) architecture, which treats GST combination weights or style embeddings as “virtual” speaking style labels within Tacotron. TP-GST learns to predict stylistic renderings from text alone, requiring neither explicit labels during training, nor auxiliary inputs for inference. We show that, when trained on an expressive speech dataset, our system can render text with more pitch and energy variation than two state-of-the-art baseline models. We further demonstrate that TP-GSTs can synthesize speech with background noise removed, and corroborate these analyses with positive results on human-rated listener preference audiobook tasks. Finally, we demonstrate that multi-speaker TP-GST models successfully factorize speaker identity and speaking style. We provide a website with audio samples 1 for each of our findings.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, methodology): “Prosody/style/emotion is the key information to improve the expressiveness of speech and the vast majority of works on expressive TTS focus on improving the prosody/style/emotion of speech [303, 377, 315, 82, 353, 318].”
Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “But the global explicit constraints can only provide an “averaged” emotional voice [10], and it is also hard to flexibly control the local emotion variations or even global intensities.”
DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Modeling these variation information [17, 3, 18, 4] can mitigate the one-to-many mapping problem and thus improve expressiveness and fidelity of synthesized speech.”
Text aware Emotional Text-to-speech with BERT (2022, background): “In one of the earliest works, TP-GST [8] learns to predict stylistic renderings from text alone, neither needs explicit labels during training but focuses majorly on improving the overall expressiveness of speech.”
A New End-to-End Long-Time Speech Synthesis System Based on Tacotron2 (2019, background): “09017 [10] Daisy Stanton, Yuxuan Wang, RJ Skerry-Ryan “Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis,” arXiv:1808.”
Expressive TTS Training With Frame and Style Reconstruction Loss (2020, methodology): “At run-time, the style token can be used to predict the speech style from text [32], or to transfer the speech style from a reference utterance to target [33].”
Robust and Fine-grained Prosody Control of End-to-end Speech Synthesis (2018, background): “Since Tacotron [1] paved the way for end-to-end Text-To-Speech (TTS) using neural networks, researchers have attempted to generate more naturally sounding speech by conditioning a TTS model via speaker and prosody embedding [2, 3, 4, 5, 6].”
An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, unknown): “This draws inspiration from Stanton et al. [158], who directly predict GSTs from text and, thus, do not require an external reference utter-ance.”
Controllable Neural Prosody Synthesis (2020, background): “Text is a strong indicator of prosodic patterns [1], but not a determinant.”
Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, methodology): “Speaker information is represented as learned speaker-wise embedding vectors, while the text information is summarized into a vector by passing the output of the Tacotron text encoder through a unidirectional RNN as done by Stanton et al. (2018). Appendix A.”
Expressive Neural Voice Cloning (2021, methodology): “To this end, we adapt the synthesis model used in Mellotron Valle et al. (2020) for the task of voice cloning.”
Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, methodology): “To generate expressive speech, reference-encoder-based methods [10, 11, 12, 13] use autoencoder-like architecture to learn latent disentangled representations and successfully factorize speaker identity and prosody; prosody prediction-based methods [5, 14] ﬁrst extract prosody attributes including…”
DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, methodology): “Text-predictedGST[21]isamethod for modeling prosody without reference audio by predicting the weight of the style token from the input text.”
Controlling Emotion Strength with Relative Attribute for End-to-End Speech Synthesis (2019, unknown): “cn acoustics [7,8], texts [9], or both [10] and sample from it with a specified condition, e.”
In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data (2019, background): “Seq2seq models also allow us to condition the model on additional observed or latent attributes that help in improving the ﬂexibility (modelling different speaker, and styles), and naturalness (Ping et al., 2017; Jia et al., 2018; Wang et al., 2018; Skerry-Ryan et al., 2018; Stanton et al., 2018).”
Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “Many approaches were recently proposed to improve modelling of expressive speech [15, 16, 17, 18].”