Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS

By Mutian He, Yan Deng, Lei He. 2019

84 citations recorded by the source corpus.

Neural TTS has demonstrated strong capabilities to generate human-like speech with high quality and naturalness, while its generalization to out-of-domain texts is still a challenging task, with regard to the design of attention-based sequence-to-sequence acoustic modeling. Various errors occur in those inputs with unseen context, including attention collapse, skipping, repeating, etc., which limits the broader applications. In this paper, we propose a novel stepwise monotonic attention method in sequence-to-sequence acoustic modeling to improve the robustness on out-of-domain inputs. The method utilizes the strict monotonic property in TTS with constraints on monotonic hard attention that the alignments between inputs and outputs sequence must be not only monotonic but allowing no skipping on inputs. Soft attention could be used to evade mismatch between training and inference. The experimental results show that the proposed method could achieve significant improvements in robustness on out-of-domain scenarios for phoneme-based models, without any regression on the in-domain naturalness test.

Subfields: Artificial Intelligence, Physiology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Physiology

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, background): “• For the alignment learning between characters/phonemes and mel-spectrograms, the works can be divided into two aspects: 1) enhancing the robustness of attention mechanism [376, 309, 297, 431, 326, 104, 38], and 2) removing attention and instead predicting duration explicitly to bridge the length m”
End-to-End Adversarial Text-to-Speech (2020, neutral): “Several mechanisms have been proposed to exploit monotonicity in tasks that require sequence alignment, including attention mechanisms [11, 20, 24, 48, 62, 72], loss functions [19, 21] and search-based approaches [29].”
Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, background): “The hybrid mechanism still has occasional alignment issues which led a number of authors to develop attention mechanisms that directly exploit monotonicity [9, 5, 6].”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, background): “Together with the soft attention mechanism, it can lead to robustness errors such as babbling, early cut-off, word repetition, and word skipping [11–14].”
A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, background): “Research has shown that this can be mitigated by enforcing monotonic alignment (He, Deng, and He 2019; Zeyer, Schl¨uter, and Ney 2021).”
Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “Note that the training target in the Conversational set is synthesized by the Tacotron 2 TTS model (see Table 1), which by itself suffers from over-generation (He et al., 2019; Zheng et al., 2019; Battenberg et al., 2020; Shen et al., 2020).”
Teacher-Student Training For Robust Tacotron-Based TTS (2019, methodology): “The techniques to improve out-of-domain performance include the GAN-based TTS framework [25] that introduces both real and generated data sequences in discriminator training, and more recently, stepwise mono-tonic attention for neural TTS [9].”
Whispered and Lombard Neural Speech Synthesis (2021, unknown): “A stepwise monotonic attention [35] is adopted for better alignment between the input and output features.”
Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit (2020, background): “To address the issue, many monotonic attention mechanisms are presented [13, 14], where only the alignment paths satisfying the monotonic condition are taken into consideration at each decoder timestep.”
Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, background): “However, these types of models are known to be less robust than traditional approaches (He et al., 2019; Zheng et al., 2019; Guo et al., 2019; Battenberg et al., 2020).”
Review of end-to-end speech synthesis technology based on deep learning (2021, background): “Since the order and position of input text and output speech in TTS task are corresponding, attention alignment in TTS is a surjective mapping from the output frames to the input tokens and should follow such strict criteria [71]:”
Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, background): “(1) The use of the Soft Attention can introduce robustness errors, such as over-generation ( e.g. , word repetitions) and under-generation ( e.g. , word skipping) [5, 11–13].”
Triple M: A Practical Text-to-Speech Synthesis System with Multi-Guidance Attention and Multi-Band Multi-Time LPCNet (2021, background): “In order to alleviate these problems, alternative attention mechanisms such as forward attention [5], GMM-based attention [6], stepwise monotonic attention [7] and dynamic convolution attention [8] are proposed.”
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “For encoder-decoder AR TTS, several previous works enforce the attention weights to be monotonic [2, 9, 36] that can effectively improve the robustness.”
Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “• Attention collapse count adds up signiﬁcant intelligibility errors [33].”
s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis (2020, background): “Various attention mechanisms are tried in the encoder-decoder attention model, like stepwise monotonic attention [10], forward-Samples are available at https://vancycici.github.io/sTransformer/ attention[11], monotonic chunkwise attention [12], and etc.”