MultiSpeech: Multi-Speaker Text to Speech with Transformer

By Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Sheng Zhao, Tao Qin. 2020

121 citations recorded by the source corpus.

Transformer-based text to speech (TTS) model (e.g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e.g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference. However, the parallel computation increases the difficulty while learning the alignment between text and speech in Transformer, which is further magnified in the multi-speaker scenario with noisy data and diverse speakers, and hinders the applicability of Transformer for multi-speaker TTS. In this paper, we develop a robust and high-quality multi-speaker Transformer TTS system called MultiSpeech, with several specially designed components/techniques to improve text-to-speech alignment: 1) a diagonal constraint on the weight matrix of encoder-decoder attention in both training and inference; 2) layer normalization on phoneme embedding in encoder to better preserve position information; 3) a bottleneck in decoder pre-net to prevent copy between consecutive speech frames. Experiments on VCTK and LibriTTS multi-speaker datasets demonstrate the effectiveness of MultiSpeech: 1) it synthesizes more robust and better quality multi-speaker voice than naive Transformer based TTS; 2) with a MutiSpeech model as the teacher, we obtain a strong multi-speaker FastSpeech model with almost zero quality degradation while enjoying extremely fast inference speed.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Sheng Zhao
7. Jinglin Liu
8. Lei He

In Model Architectures

1. Xu Tan
2. Yi Ren
3. Tao Qin
4. Zhou Zhao
5. Tie-Yan Liu
6. Jinglin Liu
7. Sheng Zhao
8. Lei He

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Sheng Zhao
5. Tao Qin
6. Jinglin Liu
7. Tie-Yan Liu
8. Rongjie Huang

In Signal Processing

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Tao Qin
5. Tie-Yan Liu
6. Jinglin Liu
7. Rongjie Huang
8. Sheng Zhao

Citations in the corpus

• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “Beyond the TTS synthesis on a single speaker, recent works [10, 11] have shown decent quality in synthesizing the speech of multiple speakers.”
• A Survey on Neural Speech Synthesis (2021, methodology): “[38] propose to restrict the attention on the source sequence into a window subset.”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “Since speaker ID (embedding) is widely used to capture speaker-level acoustic conditions in multi-speaker scenario (Chen et al., 2020), speaker embedding is used by default.”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, background): “They usually suffer from slow inference speed and robustness (word skipping and repeating) issues (Ren et al., 2019; Chen et al., 2020).”
• Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, background): “The typical TTS adaptation pipeline [1, 2, 3, 4, 5, 6, 7, 8] usually fine-tunes a well-trained multi-speaker TTS model [9, 10, 11] (source”
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation (2021, methodology): “…of the TTS models aim to synthe-size high quality speech of a single speaker from the given text (Oord et al., 2016; Wang et al., 2017; Shen et al., 2018; Ren et al., 2019; 2020) and have been extended to support multi speakers (Gibiansky et al., 2017; Ping et al., 2017; Chen et al., 2020).”
• Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis (2021, background): “Some of the current approaches [6, 7, 8, 9, 10, 11, 12] have used speaker embedding to capture the identity and speaking style of the person in the speech.”
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “…concatenating styles in place of AdaIN dramatically decreases the correlations and lowers rated naturalness, confirming our observation that all previous methods that use concatenation to incorporate style information ([1], [9], [16], [17], [19], [20], [69]) are not as effective as AdaIN.”
• Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “The state-of-art systems have an encoder-decoder structure of network with speaker embedding as conditions [9, 10, 11, 12, 13, 14, 15].”
• Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, background): “Transformer TTS suffers from unstable training, requiring tricks such as alignment constraints to work (Chen et al., 2020), and byte inputs add to the complexity.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, unknown): “All our models support the multi-speaker setting via pre-trained (Jia et al., 2018) or jointly trained speaker embeddings (Arik et al., 2017; Chen et al., 2020).”
• Hi-Fi Multi-Speaker English TTS Dataset (2021, methodology): “The dataset was intended for speaker-adaptive and multispeaker TTS [18, 19, 20, 11].”
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus (2021, unknown): “Unlike traditional TTS [3, 32–34, 44 ], there are several challenges to In the past few years, researchers work to address the challenges above in singing voices modeling, while some problems emerge: 1) SVS systems like DeepSinger [35] mine data from the web, but processed data with noise still…”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “5) Expressive FS2 [37]: The combination of both multi-speaker [7] and muli-emotion [8] FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pre-trained discriminative models to the backbone.”
• Speech Bert Embedding for Improving Prosody in Neural TTS (2021, background): “So far, in neural TTS models, the speaker and prosody embeddings are input to the decoder together with the text sequence [16, 17, 18, 19].”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “For encoder-decoder AR TTS, several previous works enforce the attention weights to be monotonic [2, 9, 36] that can effectively improve the robustness.”