Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis

By Yuxuan Wang, Daisy Stanton, Yu Zhang, R. Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, Rif A. Saurous. 2018

925 citations recorded by the source corpus.

In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system. The embeddings are trained with no explicit labels, yet learn to model a large range of acoustic expressiveness. GSTs lead to a rich set of significant results. The soft interpretable "labels" they generate can be used to control synthesis in novel ways, such as varying speed and speaking style - independently of the text content. They can also be used for style transfer, replicating the speaking style of a single audio clip across an entire long-form text corpus. When trained on noisy, unlabeled found data, GSTs learn to factorize noise and speaker identity, providing a path towards highly scalable but robust speech synthesis.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yu Zhang
6. Junichi Yamagishi
7. Jinyu Li
8. Dong Yu

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Junichi Yamagishi
4. Yu Zhang
5. Xu Tan
6. Lei He
7. Graham Neubig
8. Dong Yu

In Speech & Audio

1. Xu Tan
2. Yu Zhang
3. Wei-Ning Hsu
4. Hung-yi Lee
5. Junichi Yamagishi
6. Tao Qin
7. Lei He
8. Jinyu Li

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Junichi Yamagishi
4. Frank K. Soong
5. Tie-Yan Liu
6. Yu Zhang
7. Jinyu Li
8. Dong Yu

Citations in the corpus

• Jukebox: A Generative Model for Music (2020, methodology): “, 2011), priming (Wang et al., 2018), and variational autoencoders (Hsu et al.”
• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, methodology): “It compares GST with i-vectors for style modeling task.”
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (2018, background): “This suggests that additional care must be taken to disentangle speaker identity from prosody within the synthesis network, perhaps by integrating a prosody encoder as in [16, 24], or by training on randomly paired reference and target utterances from the same speaker.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Previous studies [62] tried to control them by learning a residual embedding.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, methodology): “The speaker encoder [2, 22] is a 6-layer convolution network,”
• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “FastSpeech 2 3.34 ± 0.11 3.85 ± 0.11 +GST (Wang et al., 2018) 3.27 ± 0.11 3.72 ± 0.11 A 3 T 3.63 ± 0.10 3.94 ± 0.11 Groundtruth 4.04 ± 0.08 4.05 ± 0.10 Table 7.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Prosody/style/emotion is the key information to improve the expressiveness of speech and the vast majority of works on expressive TTS focus on improving the prosody/style/emotion of speech [303, 377, 315, 82, 353, 318].”
• Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “We train the GST model [9] and the utterance-level emotion transfer model [18] (UET) as baseline systems for comparison.”
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech (2019, background): “Thus, the focus ofTTS research is shifting towardmore challenging tasks, such asmulti-speaker TTS [6–8], building TTS systems from small amounts of data [9], few shot voice adaptation [8, 10, 11], unsupervised prosody and speaking style modeling [12, 13], and the use of found data [13, 14].”
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron (2018, background): “(Wang et al., 2018), for example, applies a similar approach to learning representations of global style.”
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “…embedding methods have been proposed to generate diverse speech by controlling some factors such as intonation and rhythm (Skerry-Ryan et al., 2018; Wang et al., 2018), and some studies have aimed at synthesizing speech in the voices of various speakers (Jia et al., 2018; Gibiansky et al., 2017).”
• Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “Other future work includes utilizing weakly supervision to scale up training with synthetic data [21] or multitask learning [19,20], and transferring prosody and other acoustic factors from the source speech to the translated speech following [45–47].”
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, methodology): “On the other hand, Tacotron 2 GST is barely able to alter characteristics of the monotonic speaker.”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “In the field of expressive speech synthesis, several works [6], [9], [12], [27], [28], [29] have been done.”