DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders

By Yanqing Liu, Rui Xue, Lei He, Xu Tan, Sheng Zhao. 2022

35 citations recorded by the source corpus.

Current text to speech (TTS) systems usually leverage a cascaded acoustic model and vocoder pipeline with mel-spectrograms as the intermediate representations, which suffer from two limitations: 1) the acoustic model and vocoder are separately trained instead of jointly optimized, which incurs cascaded errors; 2) the intermediate speech representations (e.g., mel-spectrogram) are pre-designed and lose phase information, which are sub-optimal. To solve these problems, in this paper, we develop DelightfulTTS 2, a new end-to-end speech synthesis system with automatically learned speech representations and jointly optimized acoustic model and vocoder. Specifically, 1) we propose a new codec network based on vector-quantized auto-encoders with adversarial training (VQ-GAN) to extract intermediate frame-level speech representations (instead of traditional representations like mel-spectrograms) and reconstruct speech waveform; 2) we jointly optimize the acoustic model (based on DelightfulTTS) and the vocoder (the decoder of VQ-GAN), with an auxiliary loss on the acoustic model to predict intermediate speech representations. Experiments show that DelightfulTTS 2 achieves a CMOS gain +0.14 over DelightfulTTS, and more method analyses further verify the effectiveness of the developed system.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Yi Ren
4. Zhou Zhao
5. Tie-Yan Liu
6. Sheng Zhao
7. Jinglin Liu
8. Rongjie Huang

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Jinglin Liu
5. Tao Qin
6. Sheng Zhao
7. Rongjie Huang
8. Lei He

In Signal Processing

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Jinglin Liu
5. Tao Qin
6. Rongjie Huang
7. Sheng Zhao
8. Lei He

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “The end-to-end TTS models aim to jointly optimize the acoustic model and vocoder [4], [5].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, background): “Text-to-speech systems [2, 3, 4, 5, 6, 8, 9, 10, 11] aim to generate natural voice with both high quality and diversity.”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, unknown): “Fine-grained codec is like the frame-level feature extraction network in DelightfulTTS 2 [18], which consists of a waveform down-sampling encoder, and another mirrored up-sampling de-coder, and a residual vector quantizer acting as a feature bottle-neck.”
• Regeneration Learning: A Learning Paradigm for Data Generation (2023, unknown): “For example, in text-to-speech synthesis (Cong et al. 2021; Tan et al. 2022; Liu et al. 2022), NaturalSpeech (Tan et al. 2022) Learning Y ′ by Considering X .”