One TTS Alignment to Rule Them All

By Rohan Badlani, Adrian La'ncucki, Kevin J. Shih, Rafael Valle, Wei Ping, Bryan Catanzaro. 2021

106 citations recorded by the source corpus.

Speech-to-text alignment is a critical component of neural text-to-speech (TTS) models. Autoregressive TTS models typically use an attention mechanism to learn these alignments on-line. However, these alignments tend to be brittle and often fail to generalize to long utterances and out-of-domain text, leading to missing or repeating words. Most non-autoregressive end-to-end TTS models rely on durations extracted from external sources. In this paper we leverage the alignment mechanism proposed in RAD-TTS and demonstrate its applicability to wide variety of neural TTS models. The alignment learning framework combines the forward-sum algorithm, Viterbi algorithm, and an efficient static prior. In our experiments, the framework improves all tested TTS architectures, both autoregressive (Flowtron, Tacotron 2) and non-autoregressive (FastPitch, FastSpeech 2, RAD-TTS). Specifically, it improves alignment convergence speed, simplifies the training pipeline by eliminating need for external aligners, enhances robustness to errors on long utterances and improves the perceived speech synthesis quality, as judged by human evaluators.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sheng Zhao
2. Dong Yu
3. Yanqing Liu
4. Dan Su
5. Songxiang Liu
6. Sung Ju Hwang
7. Jinyu Li
8. D. Lim

Citations in the corpus

• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “For aligner, we use the same architecture and loss functions from the original paper [20].”
• RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion (2022, methodology): “To get the ground-truth duration, we use an online alignment module jointly trained with our model using an alignment loss L align [20].”
• JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech (2022, methodology): “In this work, we incorporate an alignment learning framework [18] into our joint training framework for obtaining the required token duration d during the training on the fly.”
• Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosody (2023, background): “Additionally, it incorporates a robust alignment framework [28], leading to fast convergence and thus allowing for quick experimentation.”
• EdiTTS: Score-based Editing for Controllable Text-to-Speech (2021, background): “We believe ongoing advancements in robust TTS alignments [26] will help improve not only EdiTTS, but also nonautoregressive TTS pipelines in general.”
• M2-CTTS: End-to-End Multi-Scale Multi-Modal Conversational Text-to-Speech Synthesis (2023, methodology): “unsupervised duration modeling [17] rather than using supervised duration model with external aligner, such as Montreal Forced Aligner [18], because external aligners have risk of out-of-distribution problem and soft alignment has more flexibility for expressive performance.”
• DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs (2022, background): “They either leverage an external text-to-acoustic alignment module (Ren et al., 2019; 2021a; Peng et al., 2020; Elias et al., 2021) or jointly train one within the TTS model (Zeng et al., 2020; Miao et al., 2021; Badlani et al., 2021).”