Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling

By Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, Yonghui Wu. 2020

116 citations recorded by the source corpus.

This paper presents Non-Attentive Tacotron based on the Tacotron 2 text-to-speech model, replacing the attention mechanism with an explicit duration predictor. This improves robustness significantly as measured by unaligned duration ratio and word deletion rate, two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model. With the use of Gaussian upsampling, Non-Attentive Tacotron achieves a 5-scale mean opinion score for naturalness of 4.41, slightly outperforming Tacotron 2. The duration predictor enables both utterance-wide and per-phoneme control of duration at inference time. When accurate target durations are scarce or unavailable in the training data, we propose a method using a fine-grained variational auto-encoder to train the duration predictor in a semi-supervised or unsupervised manner, with results almost as good as supervised training.

Subfields: Artificial Intelligence, Biomedical Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Yu Zhang
5. Ye Jia
6. Jinyu Li
7. Shinji Watanabe
8. Sheng Zhao

In Biomedical Engineering

1. Ye Jia
2. Michelle Tadmor Ramanovich
3. Heiga Zen
4. Mohammad Norouzi
5. Nima Mesgarani
6. Cong Han
7. Xu Tan

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Jinyu Li
5. Yu Zhang
6. Shinji Watanabe
7. Frank K. Soong
8. Nima Mesgarani

Citations in the corpus

• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Non-attentive Tacotron [57] extends this by making σ i trainable, but the trained parameters introduce instability for E2E training with adversarial loss, similar to issues of attention-based upsamplers.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Replacing Attention with Duration FastSpeech [284], DurIAN [411], EATS [67] Glow-TTS [156], [190, 19, 355, 422, 73, 298]”
• Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “As a baseline we used the Tacotron 2 model [6] with some small modifications: specifically it used Gaussian mixture model (GMM) attention [40, 41] and a reduction factor of 2 [2], both of which have been shown to improve the robustness [24].”
• CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, positive): “Thanks to the duration-based autoregressive speech generation (Shen et al., 2020) used in both the PnG NAT TTS model and the Translatotron 2 S2ST model.”
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS (2021, methodology): “NAT w/ PnG BERT 4.47 ± 0.05 0.28 ± 0.05 0.15 ± 0.06 w/o consistent masking 4.44 ± 0.05 0.14 ± 0.06 0.10 ± 0.06 w/o word alignment 4.45 ± 0.05 0.15 ± 0.06 0.12 ± 0.06 w/o pre-training 4.30 ± 0.06 0.12 ± 0.05 0.06 ± 0.05”
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis (2021, methodology): “To make the structure non-autoregressive and speed-up inference, we adopt the Gaussian upsampling introduced in the non-attentive Tacotron [9].”
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, methodology): “Before comparing systems quantitatively, we showcase some qualitative properties of Tacotron 2 and Transformer TTS. Figure 3 compares autoregressive models on their alignments.”
• Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “Tacotron-like autoregressive models further factorize this distribution into ∏ p(xi | x i, c).”
• Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “We used a multilingual multispeaker PnG NAT model [51, 52] to synthesize the source text from the MT dataset into speech, and used the same English TTS model as in [9] to synthesize the targets.”
• Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation (2022, methodology): “[1] Yuxuan Wang, RJ Skerry-Ryan, et al., “Tacotron: Towards end-to-end speech synthesis,” in Proc.”
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, background): “(1) The use of the Soft Attention can introduce robustness errors, such as over-generation ( e.g. , word repetitions) and under-generation ( e.g. , word skipping) [5, 11–13].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Since one-to-many regression problems like TTS can benefit from autoregressive decoding, it is also possible to combine the autoregressive method with duration predictor to further improve the stability of TTS models, such as the alignment methods used in DurIAN, Non-Attentive Tacotron [190], Device”
• Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “On the Fisher set, the UDR from Translatotron 2 is about 7 times lower than from the original Translatotron, and is about the same as the training target.”
• HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, background): “In the first stage, low-resolution intermediate representations (e.g., linguistic features, mel-spectrograms) are predicted from text data (Li et al., 2020; Shen et al., 2018; 2020).”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “As for the duration alignment, WaveGrad 2 [8] generates duration information with a non-attentive Tacotron [97] and optimizes a duration predictor towards ground-truth duration during training.”
• The Multi-Speaker Multi-Style Voice Cloning Challenge 2021 (2021, unknown): “Non-Attentive Tacotron [25] is similar to DurIAN, which introduces Gaussian upsampling method significantly improving the naturalness compared to the vanilla upsampling through repetition.”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “Instead, an L 2 loss on the total predicted duration of the entire utterance is used (similar to the “na¨ıve approach” of unsupervised duration modeling in Shen et al. (2020)).”