SpeedySpeech: Efficient Neural Speech Synthesis

By Jan Vainer, Ondrej Dusek. 2020

52 citations recorded by the source corpus.

While recent neural sequence-to-sequence models have greatly improved the quality of speech synthesis, there has not been a system capable of fast training, fast inference and high-quality audio synthesis at the same time. We propose a student-teacher network capable of high-quality faster-than-real-time spectrogram synthesis, with low requirements on computational resources and fast training time. We show that self-attention layers are not necessary for generation of high quality audio. We utilize simple convolutional blocks with residual connections in both student and teacher networks and use only a single attention layer in the teacher model. Coupled with a MelGAN vocoder, our model's voice quality was rated significantly higher than Tacotron 2. Our model can be efficiently trained on a single GPU and can run in real time even on a CPU. We provide both our source code and audio samples in our GitHub repository.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Sheng Zhao
6. Zhou Zhao
7. Jinglin Liu
8. Frank K. Soong

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Jinglin Liu
7. Sheng Zhao
8. Frank K. Soong

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Replacing Attention with Duration FastSpeech [284], DurIAN [411], EATS [67] Glow-TTS [156], [190, 19, 355, 422, 73, 298]”
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech (2021, methodology): “[17] J. Vainer and O. Dušek, “Speedyspeech: Efficient neural speech synthesis,” arXiv preprint arXiv:2008.03802, 2020.”
• SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model (2021, methodology): “Our convolutional residual encoder is based on [15], however we used the Mish [17] instead of ReLU activation function.”
• Fcl-Taco2: Towards Fast, Controllable and Lightweight Text-to-Speech Synthesis (2021, methodology): “On the contrary, NAR models [12-21] achieve extremely fast inference speed, where a length regulator is used to replace the attention module in AR models to predict phoneme durations and expand the phoneme embeddings for parallel generation.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Nekvinda T, Dušek O (2020) One model, many languages: Meta-learning for multilingual text-tospeech. arXiv preprint arXiv:200800768 148.”
• Denoispeech: Denoising Text to Speech with Frame-Level Noise Modeling (2020, methodology): “We use the sum of mean absolute error (MAE) and mean structural similarity (MSSIM) [15] losses for mel-spectrogram value prediction inspired by [16].”
• Revisiting Over-Smoothness in Text to Speech (2022, methodology): “Non-autoregressive text to speech (NAR-TTS) models (Ren et al., 2019, 2020; Peng et al., 2020; ∗ Corresponding author Vainer and Dušek, 2020; Łańcucki, 2020; Kim et al., 2020; Miao et al., 2020) have shown much faster inference speed than their autoregressive counterparts (Wang et al., 2017; Shen…”
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech (2021, methodology): “In addition to improving the performance of non-autoregressive models, some works focus on lightweight and portable model designs: SpeedySpeech [32] replaces the self-attention layers with fully convolutional blocks to reduce the computational complexity.”
• Expressive Text-to-Speech Using Style Tag (2021, methodology): “The entire structure is based on non-autoregressive feedforward TTS models [3, 4, 8], and end-to-end trainable by jointly optimizing duration information with the entire system.”