Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

By Jaehyeon Kim, Jungil Kong, Juhee Son. 2021

1,342 citations recorded by the source corpus.

Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Jinyu Li
5. Dong Yu
6. Shinji Watanabe
7. Wei-Ning Hsu
8. Zhou Zhao

In Generative Models

1. Yi Ren
2. Zhou Zhao
3. Tie-Yan Liu
4. Yoshua Bengio
5. Xu Tan
6. Tao Qin
7. Aaron C. Courville
8. Sheng Zhao

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Wei-Ning Hsu
4. Sheng Zhao
5. Yi Ren
6. Zhou Zhao
7. Shinji Watanabe
8. Tao Qin

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Dong Yu
4. Shinji Watanabe
5. Tao Qin
6. Tie-Yan Liu
7. Nima Mesgarani
8. Wei-Ning Hsu

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Specifically, we use a model trained with ground truth human ratings obtained using ITU-T P.808 [81] 6 The DNSMOS score predicted in the range of [1, 5], with a larger value indicating better quality of the input speech.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Previous works using raw waveform [8, 9] and mel-spectrogram [1, 2, 10, 7, 11] as data representations suffer from these intricate complexities during speech generation.”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In the garden, where the flowers that the gardener who retired last year still bloomed, the children who play there every afternoon find peace and joy.”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “Recently, deep generative AI has achieved remarkable results in various tasks, leading to the emergence of many transformative real-world applications (Brown et al., 2020; Ramesh et al., 2022; Ho et al., 2020; Rombach et al., 2022; Borsos et al., 2023; Kim et al., 2021; Chiang et al., 2019).”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “However, these models encode style in a low-dimensional space and impose an overly simple distribution of speech given text and residual embedding [51, 56].”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Some previous works have made significant progress towards human-level performance [3, 4, 5].”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “This formulation, with continuous latent variables and training an expressive prior using normalizing flows has been quite successful for speech synthesis [17, 36].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “The predicted mel-spectrograms are transformed into audio samples using pre-trained HiFi-GAN V1 8 [31].”
• PolyVoice: Language Models for Speech to Speech Translation (2023, background): “In recent years, neural text-to-speech (TTS) synthesis has achieved significant developments, and the progress of neural network structure makes continuous improvements in the intelligence of synthetic speech (Wang et al., 2017; Ren et al., 2019; Kim et al., 2021; Popov et al., 2021).”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, background): “Text-to-speech systems [2, 3, 4, 5, 6, 8, 9, 10, 11] aim to generate natural voice with both high quality and diversity.”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “Beyond the TTS synthesis on a single speaker, recent works [10, 11] have shown decent quality in synthesizing the speech of multiple speakers.”
• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Although ideally having advantages over two-stage systems, different end-to-end models face their own challenges and limitations: 1) no significant improvement over two-stage models in terms of voice quality, such as FastSpeech 2s [3], EATS [6], and WAVE-TACOTRON [8]; 2) slow inference speed due to ”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, methodology): “Then we compare VQTTS with other current popular methods, including Tacotron 2, GlowTTS, FastSpeech 2 and the fully end-to-end TTS model VITS [38].”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, positive): “Our results in terms of MOS match the ones reported by the VITS article (Kim et al., 2021).”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “End-to-end TTS models have recently been proposed, such as Ren et al. (2021), Donahue et al. (2021), Weiss et al. (2021), Kim et al. (2021), and Chen et al. (2021b).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Model One-Stage Training AR/NAR Modeling Architecture Char2Wav [309] N AR Seq2Seq RNN ClariNet [263] N AR Flow CNN FastSpeech 2s [286] Y NAR GAN Self-Att/CNN EATS [67] Y NAR GAN CNN Wave-Tacotron [379] Y AR Flow CNN/RNN/Hybrid EfficientTTS-Wav [229] Y NAR GAN CNN VITS [157] Y NAR VAE+Flow CNN/Self-A”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, background): “Nowadays, the state-of-the-art TTS systems [1], [2], [3] are able to produce natural and high-quality speech.”