Parallel Tacotron: Non-Autoregressive and Controllable TTS

By Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron J. Weiss, Yonghui Wu. 2020

110 citations recorded by the source corpus.

Although neural end-to-end text-to-speech models can synthesize highly natural speech, there is still room for improvements to its efficiency and naturalness. This paper proposes a non-autoregressive neural text-to-speech model augmented with a variational autoencoder-based residual encoder. This model, called Parallel Tacotron, is highly parallelizable during both training and inference, allowing efficient synthesis on modern parallel hardware. The use of the variational autoencoder relaxes the one-to-many mapping nature of the text-to-speech problem and improves naturalness. To further improve the naturalness, we use lightweight convolutions, which can efficiently capture local contexts, and introduce an iterative spectrogram loss inspired by iterative refinement. Experimental results show that Parallel Tacotron matches a strong autoregressive baseline in subjective evaluations with significantly decreased inference time.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Lei He
5. Frank K. Soong
6. Sheng Zhao
7. Yu Zhang
8. Jinyu Li

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Lei He
4. Sheng Zhao
5. Jinyu Li
6. Tie-Yan Liu
7. Yu Zhang
8. Frank K. Soong

Citations in the corpus

• MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “The upsampling is done by copying the original text embedding to the target length of specified duration with positional embeddings to capture frame positions within text units as described in [22].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “Generation Methods in TTS. Previous works have demonstrated that NAR-based models [3, 33, 34, 7, 5, 11] enjoy better robustness and generation speed than AR-based models, because they explicitly model the duration and predict all features simultaneously.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Since TTS is a typical sequence generation task and can be modeled through typical generative models, we can categorize in terms of different generative models: normal sequence generation model, flow, GAN, VAE, and diffusion model.”
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “, 2019) and Parallel Tacotron (Elias et al., 2020) substantially improved inference speed and pronunciation robustness by utilizing nonautoregressive architectures and building hard monotonic alignments from estimated token lengths.”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “With the help of utterance-level prosody vector, we do not need an autoregressive prosody predictor as [23] for faster inference.”
• Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “Parallel Tacotron is another brilliant invention in recent times for neural TTS approach proposed by I. Elias et al. [35].”
• Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, methodology): “Moreover, neural TTS tends to generate many errors such as too short or repeated speech [29], and multi-speaker TTS To address the above two problems, we introduce a discrete representation to be used for both ASR and TTS. Theo-retically, the unreal features which do not exist in real features are…”
• DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs (2022, background): “They either leverage an external text-to-acoustic alignment module (Ren et al., 2019; 2021a; Peng et al., 2020; Elias et al., 2021) or jointly train one within the TTS model (Zeng et al., 2020; Miao et al., 2021; Badlani et al., 2021).”
• Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “It is tempting to use a non-autoregressive architecture for the synthesizer of Translatotron 2, which may significantly reduce its inference latency, similar to recent works on non-autoregressive TTS (Ren et al., 2019; 2021; Guo et al., 2021; Lee et al., 2021b; Elias et al., 2021b;a).”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “For example, TalkNet [12] uses the CTC-based automatic speech recognition (ASR) model Quartznet [111], FastSpeech 2 [174] uses the forcedalignment tool MFA toolkit [135], DurIAN [243] uses a external alignment model [51, 252], RobuTrans [119] uses speech recognition tools, Parallel Tacotron [49] and”
• Axial Residual Networks for CycleGAN-based Voice Conversion (2021, background): “First proposed in the domain of natural language processing, it has been successfully adopted in processing audio information [23, 24]; when applied to the domain of audio, the inductive bias of the 1D lightweight convolution is that information in nearby frequency bands can be effectively processed”
• Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “C N ON - AUTOREGRESSIVE SYNTHESIZER Following recent non-autoregressive TTS works (Ren et al., 2021; Guo et al., 2021; Lee et al., 2021; Elias et al., 2021a), we explored using non-autoregressive synthesizer in Translatotron 2, which may have significantly lower latency at inference time.”
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, methodology): “Parallel Tacotron [14] requires supervised token durations to be provided by an external aligner.”