Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

By Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang, Xu Tan. 2019

233 citations recorded by the source corpus.

This paper introduces a new end-to-end text-to-speech (E2E-TTS) toolkit named ESPnet-TTS, which is an extension of the open-source speech processing toolkit ESPnet. The toolkit supports state-of- the-art E2E-TTS models, including Tacotron 2, Transformer TTS, and FastSpeech, and also provides recipes inspired by the Kaldi automatic speech recognition (ASR) toolkit. The recipes are based on the design unified with the ESPnet ASR recipe, providing high reproducibility. The toolkit also provides pre-trained models and samples of all of the recipes so that users can use it as a baseline. Furthermore, the unified design enables the integration of ASR functions with TTS, e.g., ASR-based objective evaluation and semi- supervised learning with both ASR and TTS models. This paper describes the design of the toolkit and experimental evaluation in comparison with other toolkits. The experimental results show that our models can achieve state-of-the-art performance comparable to the other latest toolkits, resulting in a mean opinion score (MOS) of 4.25 on the LJSpeech dataset. The toolkit is publicly available at https://github.com/espnet/espnet.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Wei-Ning Hsu
5. Yossi Adi
6. Juan Pino
7. Changhan Wang
8. Shinji Watanabe

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yossi Adi
5. Junichi Yamagishi
6. Wei-Ning Hsu
7. Shinji Watanabe
8. Hieu-Thi Luong

Citations in the corpus

• ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “We applied an open-source code [40] 8 for fair comparison with other speech samples provided.”
• Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input (2021, methodology): “The implementation we use [18](2), trained on the LJ Speech Dataset [19], takes characters as input and converts them to phonemes.”
• KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset (2021, methodology): “We used ESPnet-TTS toolkit [22] to build the E2E-TTS models based on Tacotron 2 [7] and Transformer [9] architectures.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “ESPnet (Watanabe et al., 2018; Hayashi et al., 2020), NeMo, and OpenSeq2Seq (Kuchaiev et al., 2018b) are the most similar toolkits that also support multiple tasks.”
• High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency (2020, methodology): “For comparison we selected the ESPnet toolkit [28] because it provides many state-of-the-art pretrained models.”
• Speech Recognition via Large-Scale Weak Supervision (unknown): “ESPnet (Watanabe et al. 2018, Hayashi et al. 2020).”
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis (2020, methodology): “, 2020) as the vocoder since they are popular in speech synthesis (Hayashi et al., 2020; Ren et al., 2020a; Blaauw & Bonada, 2020; Lu et al., 2020) to ensure fast training and inference speed and also high quality.”
• A survey on speech synthesis techniques in Indian languages (2020, background): “[166], systems provides better functionalities under different languages and replaces many of the available conventional TTS systems.”
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022 (2022, unknown): “The main track uses the BVCC dataset [6], which contains data obtained by conducting large-scale listening tests on samples from 187 different systems from previous Blizzard Challenges, Voice Conversion Challenges, and ESPnet-TTS [7] published samples.”
• Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously (2023, methodology): “For the TTS models, we use the pre-trained checkpoints from ESPNet-TTS[10] and directly apply its demo code to synthesize all the samples.”
• NAUTILUS: A Versatile Voice Cloning System (2020, methodology): “v3, implemented with ESPnet [76], as the initial model.”
• ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “ESPnet was originally developed for the ASR task (Watanabe et al., 2018), and recently extended to the text-to-speech (TTS) task (Hayashi et al., 2020).”
• JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment (2020, methodology): “For comparative experiments, we train another three models using ESPnet-TTS [13] which is open-source speech processing toolkit supporting state-of-the-art end-to-end TTS models.”