Representation Mixing for TTS Synthesis

By Kyle Kastner, João Felipe Santos, Yoshua Bengio, Aaron C. Courville. 2018

46 citations recorded by the source corpus.

Recent character and phoneme-based parametric TTS systems using deep learning have shown strong performance in natural speech generation. However, the choice between character or phoneme input can create serious limitations for practical deployment, as direct control of pronunciation is crucial in certain cases. We demonstrate a simple method for combining multiple types of linguistic information in a single encoder, named representation mixing, enabling flexible choice between character, phoneme, or mixed representations during inference. Experiments and user studies on a public audiobook corpus show the efficacy of our approach.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Lei He
7. Yu Zhang
8. Heiga Zen

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “It is worth to mention that there are some works [264, 262, 151] using mixed representations of characters and phonemes as input to address the data sparsity problem.”
• Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, background): “Separately, some authors [10] have moved back toward the purely location-based GMM attention introduced by Graves in [1], and some have proposed stabilizing GMM attention by using soft-plus nonlinearities in place of the exponential function [11, 12].”
• Fastpitch: Parallel Text-to-Speech with Pitch Prediction (2020, methodology): “We tried the mixed approach of training simultaneously on phonemes and characters [9].”
• FeatherTTS: Robust and Efficient attention based Neural TTS (2020, methodology): “Such as Tacotron [1] uses the purely contentbased attention mechanism introduced in [13], Tacotron2 [2] uses an improved hybrid location-sensitive mechanism introduced in [14], some works [8, 15, 16] explore the use of monotonic attention mechanisms, and some authors [17, 18] use the location-based ”
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit (2020, background): “ress the issue, many monotonic attention mechanisms are presented [13, 14], where only the alignment paths satisfying the monotonic condition are taken into consideration at each decoder timestep. In [12, 15], the location-based GMM attention introduced in [16] is also studied in TTS systems to gene”
• Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning (2019, methodology): “End-to-end TTS models have typically used character [2] or phoneme [8,23] input representations, or hybrids between them [24, 25].”
• Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “• Many seq2seq methods introduce phoneme input [14–16, 19, 23–35] rather than character input [10–22].”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, background): “Some works [22, 23] attempt to enhance the TTS phoneme encoder with character information directly rather than introducing an auxiliary BERT model.”