JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech

By D. Lim, Sunghee Jung, Eesung Kim. 2022

75 citations recorded by the source corpus.

In neural text-to-speech (TTS), two-stage system or a cascade of separately learned models have shown synthesis quality close to human speech. For example, FastSpeech2 transforms an input text to a mel-spectrogram and then HiFi-GAN generates a raw waveform from a mel-spectogram where they are called an acoustic feature generator and a neural vocoder respectively. However, their training pipeline is somewhat cumbersome in that it requires a fine-tuning and an accurate speech-text alignment for optimal performance. In this work, we present end-to-end text-to-speech (E2E-TTS) model which has a simplified training pipeline and outperforms a cascade of separately learned models. Specifically, our proposed model is jointly trained FastSpeech2 and HiFi-GAN with an alignment module. Since there is no acoustic feature mismatch between training and inference, it does not requires fine-tuning. Furthermore, we remove dependency on an external speech-text alignment tool by adopting an alignment learning objective in our joint training framework. Experiments on LJSpeech corpus shows that the proposed model outperforms publicly available, state-of-the-art implementations of ESPNet2-TTS on subjective evaluation (MOS) and some objective evaluations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Cong Han
3. Sheng Zhao
4. Nima Mesgarani
5. Yanqing Liu
6. Jinyu Li
7. Tao Qin
8. Gavin Mischler

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Previous works using raw waveform [8, 9] and mel-spectrogram [1, 2, 10, 7, 11] as data representations suffer from these intricate complexities during speech generation.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Our baseline models consisted of the three highest-performing public models: VITS [3], StyleTTS [6], and JETS [32] for LJSpeech; and VITS, YourTTS [60], and StyleTTS for LibriTTS.”
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “A similar discrepancy has been reported in a very recent study that examines the effects of evaluation methods on the MOS results [62], and our evaluation of VITS is comparable to other studies that have tried to reproduce it on both LJSpeech and LibriTTS datasets [63], [64], [65], [66].”