Mellotron: Multispeaker Expressive Voice Synthesis by Conditioning on Rhythm, Pitch and Global Style Tokens

By Rafael Valle, Jason Li, Ryan J. Prenger, Bryan Catanzaro. 2019

165 citations recorded by the source corpus.

Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data. By explicitly conditioning on rhythm and continuous pitch contours from an audio signal or music score, Mellotron is able to generate speech in a variety of styles ranging from read speech to expressive speech, from slow drawls to rap and from monotonous voice to singing voice. Unlike other methods, we train Mellotron using only read speech data without alignments between text and audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We provide F0 Frame Errors and synthesized samples that include style transfer from other speakers, singers and styles not seen during training, procedural manipulation of rhythm and pitch and choir synthesis.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Frank K. Soong
6. Yi Ren
7. Yossi Adi
8. Rongjie Huang

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Yossi Adi
7. Jinglin Liu
8. Rongjie Huang

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “2) Extract the pitch and energy information from speech and extract duration from paired text and speech data [284, 286, 177, 155, 233, 359].”
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (2020, methodology): “In Mellotron, utterances are conditioned on both audio statistics and a latent embedding of acoustic features derived from a reference acoustic representation.”
• Fastpitch: Parallel Text-to-Speech with Pitch Prediction (2020, background): “Conditioning on fundamental frequency with voiced/unvoiced decisions is a common approach to augmenting the model with singing capabilities [24, 13], or adaptability to other speakers [11].”
• Efficient neural speech synthesis for low-resource languages through multilingual modeling (2020, methodology): “The mel-spectrogram features were then decoded by a WaveGlow vocoder [23], that was trained in universal fashion [24] on a proprietary data set consisting of 5000 sentences each from 3 female and 2 male speakers.”
• Expressive Neural Voice Cloning (2021, methodology): “Following the notation used in [26], we call this latent variable rhythm , since it controls the timing aspects of synthesized speech.”
• Towards Universal Text-to-Speech (2020, methodology): “Recent years, end-to-end (E2E) models have been widely used in speech synthesis [6, 7, 8, 9, 10, 11, 12], where the models can be directly trained on text-speech pairs with minimal engineering efforts.”
• Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis (2021, background): “, [17, 18, 19]), but as these representations are discovered, rather than explicitly formulated, they often lack a priori interpretability (though post hoc listening often reveals some uniform perceptual quality).”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, methodology): “, 2018b) introduces the idea of global style token (GST) and its derivative (Valle et al., 2020) further manages to memorize and reproduce the global scale style features of speech.”
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech (2021, unknown): “• Mellotron [29]: Tacotron2 [2] based autoregressive GST model, with ability to condition pitch and duration explicitly.”
• AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines (2020, background): “, pitch/energy for supervised settings [8] and learned embeddings for unsupervised variants [9, 10, 11]) as augmented input besides normalized texts.”
• Zero-Shot Text-to-Speech Synthesis Conditioned Using Self-Supervised Speech Representation Model (2023, methodology): “Two main approaches have been used to obtain such embeddings: speaker-recognition-based embedding such as d-vector [4, 5] and x-vector [3, 6], and embedding from the reference encoder trained simultaneously with the acoustic model, such as using global style tokens (GSTs) [7, 8].”
• STYLETTS-VC: ONE-SHOT VOICE CONVERSION BY KNOWLEDGE TRANSFER FROM STYLE-BASED TTS MODELS (2022, background): “VC systems such as Mellotron [15] and Cotatron [16], on the other hand, use phoneme alignment and pitch curve from the source speech and re-synthesize the speech of the target speaker.”
• Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages (2020, methodology): “In addition to GST network, we adopt scheme in [25] to explicitly model expressive speech variables, such as fundamental frequency contour (F0) or pitch, and voicing decision (voiced/unvoiced), and rhythm variables.”
• Hi-Fi Multi-Speaker English TTS Dataset (2021, methodology): “The LibriTTS is often used for research in low resource multi-speaker speech synthesis [16, 17] and voice adaptation [11].”
• Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features (2019, background): “[17] propose “Mellotron” that explicitly conditions the TC2 model on continuous rhythm and pitch contours to achieve fine-grained control of rhythm and pitch of synthesized speech.”
• Unsupervised Cross-Domain Singing Voice Conversion (2020, background): “Mellotron [21], presented a sequence to sequence architecture conditioned on text and pitch to synthesize singing, without training on a singing dataset.”