Uncovering Latent Style Factors for Expressive Speech Synthesis

By Yuxuan Wang, R. Skerry-Ryan, Ying Xiao, Daisy Stanton, Joel Shor, Eric Battenberg, R. Clark, Rif A. Saurous. 2017

53 citations recorded by the source corpus.

Prosodic modeling is a core problem in speech synthesis. The key challenge is producing desirable prosody from textual input containing only phonetic information. In this preliminary study, we introduce the concept of "style tokens" in Tacotron, a recently proposed end-to-end neural speech synthesis model. Using style tokens, we aim to extract independent prosodic styles from training data. We show that without annotation data or an explicit supervision signal, our approach can automatically learn a variety of prosodic variations in a purely data-driven way. Importantly, each style token corresponds to a fixed style factor regardless of the given text sequence. As a result, we can control the prosodic style of synthetic speech in a somewhat predictable and globally consistent way.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Junichi Yamagishi
2. Xu Tan
3. Tao Qin
4. Frank K. Soong
5. Tie-Yan Liu
6. Xin Eric Wang
7. Ron J. Weiss
8. Lei He

Citations in the corpus

• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron (2018, methodology): “In this work, we augment Tacotron with explicit prosody controls.”
• Forward–Backward Decoding Sequence for Regularizing End-to-End TTS (2019, methodology): “The output of the final CNN layer is fed into a single bidirectional [19] LSTM [20] layer to generate the hidden representation of input sequence.”
• GraphTTS: Graph-to-Sequence Modelling in Neural Text-to-Speech (2020, methodology): “Some researchers make some progress to use a reference encoder to capture prosody information from audios by several feature learning techniques [4][5][6][7][8].”
• From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint (2020, background): “proposed the style tokens to uncover the latent space regarding speech attributes that are hard to define and label [6, 7].”
• Bi-Level Speaker Supervision for One-Shot Speech Synthesis (2020, methodology): “With the development of deep learning, end-to-end speech synthesis models, such as Tacotron [1] and its varieties [2–4], are proposed to simplify traditional TTS pipeline [5–8] with a single neural network.”
• Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder (2018, background): “The most relevant research related to ours might be [19, 12], which proposed the seq2seq SS models that can learn and control speaking styles in an unsupervised manner.”
• Investigation of Fast and Efficient Methods for Multi-Speaker Modeling and Speaker Adaptation (2021, background): “However, this heavy data requirement can’t always be satisfied for many personalized applications, which can only afford a handful training data [4, 5].”
• Emotional Speech Synthesis with Rich and Granularized Control (2019, background): “, Tacotron), the expressiveness of synthesized speech is controlled by conditioning additional embedding vectors that implicitly provide prosodyrelated latent features typically generated by another neural network [8, 9, 10, 11, 12, 13].”
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, methodology): “This extends and improves on preliminary work presented by the same group in [33], which learned framewise rather than utterance-level control.”