Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis

By Eric Battenberg, R. Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, Tom Bagby. 2019

126 citations recorded by the source corpus.

Despite the ability to produce human-level speech for in-domain text, attention-based end-to-end text-to-speech (TTS) systems suffer from text alignment failures that increase in frequency for out-of-domain text. We show that these failures can be addressed using simple location-relative attention mechanisms that do away with content-based query/key comparisons. We compare two families of attention mechanisms: location-relative GMM-based mechanisms and additive energy-based mechanisms. We suggest simple modifications to GMM-based attention that allow it to align quickly and consistently during training, and introduce a new location-relative attention mechanism to the additive energy-based family, called Dynamic Convolution Attention (DCA). We compare the various mechanisms in terms of alignment speed and consistency during training, naturalness, and ability to generalize to long utterances, and conclude that GMM attention and DCA can generalize to very long utterances, while preserving naturalness for shorter, in-domain utterances.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Yu Zhang
7. Heiga Zen
8. Sheng Zhao

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Lei He
8. Yu Zhang

Citations in the corpus

• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, positive): “Dynamic convolution attention parameters follow the original paper [25].”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “We thus expose BASE TTS to speech segments which are between 0 and 40 seconds long, containing one or more sentences, so that the model becomes robust to both very short inputs, and learns from longer context [55].”
• A Survey on Neural Speech Synthesis (2021, background): “WaveGAN [396] Voc melS FF−→wav ICASSP20 2019.10 DCA-Tacotron [17] AM ph AR−→melS AR−→wav ICASSP20 2019.10 WaveFlow [265] Voc melS AR−→wav ICML20 2019.12 SqueezeWave [426] Voc melS FF−→wav arXiv20 2020.01 AlignTTS [422] AM ch/ph FF−→melS FF−→wav ICASSP20 2020.03 RobuTrans [190] AM ph AR−→melS AR−→wav”
• Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “We adopt the robust GMM attention [24] to connect the encoder and the decoder.”
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, methodology): “Similar to (Battenberg et al., 2019), we extract 227 utterances from the first two chapters of the book, Harry Potter and the Philosopher’s Stone .”
• End-to-End Adversarial Text-to-Speech (2020, background): “For TTS, incorporating this constraint has been found to help generalisation to long sequences (Battenberg et al., 2020).”
• Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, background): “Together with the soft attention mechanism, it can lead to robustness errors such as babbling, early cut-off, word repetition, and word skipping [11–14].”
• Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training (2020, methodology): “The proposed architecture contains a CBHG-based text encoder [22], an auto-regressive decoder with GMM-based attention [23], the domain adversarial training module, and the speaker representation module.”
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention (2021, background): “, 2019) location-relative attention (Battenberg et al., 2020).”
• Triple M: A Practical Text-to-Speech Synthesis System with Multi-Guidance Attention and Multi-Band Multi-Time LPCNet (2021, background): “In order to alleviate these problems, alternative attention mechanisms such as forward attention [5], GMM-based attention [6], stepwise monotonic attention [7] and dynamic convolution attention [8] are proposed.”
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, background): “However, these types of models are known to be less robust than traditional approaches (He et al., 2019; Zheng et al., 2019; Guo et al., 2019; Battenberg et al., 2020).”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[10] proposed Dynamic Convolution Attention (DCA), which removed content-based termsWsi and V hj , leaving only location-sensitive term Ufi,j as static filters, while adding a set of learned dynamic filters Tgi,j and a single fixed prior filter pi,j .”
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency (2020, methodology): “A significant hindrance for many models is that the inference time depends on the sentence length, which can lead to big slowdowns especially now that very long sentence generation is made possible through novel alignment models [19].”
• Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, methodology): “(2018) and the GMMv2b attention mechanism from Battenberg et al. (2019). For the heuristic models compared in Section 2.”
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit (2020, background): “lignment are not fully utilized, the alignment procedure lacks robustness in inference, which leads to skipping or repeating words, incomplete synthesis, or an inability to synthesize long utterances [12]. To address the issue, many monotonic attention mechanisms are presented [13, 14], where only t”
• Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement (2020, methodology): “Firstly, GMMbased attention [20] is applied for improving the stability of the synthesis and reducing bad cases.”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, background): “There has been evidence of instability in this attention-based alignment procedure, as it may introduce missing or repeating words, incomplete synthesis, or an inability to generalize to longer utterances [24].”
• Fast and Lightweight On-Device TTS with Tacotron2 and LPCNet (2020, methodology): “Finally, to improve synthesis of long sentences we replace location-sensitive attention with dynamic convolutional attention [18].”