High quality, lightweight and adaptable TTS using LPCNet

By Zvi Kons, Slava Shechtman, Alexander Sorin, Carmel Rabinovitz, Ron Hoory. 2019

55 citations recorded by the source corpus.

We present a lightweight adaptable neural TTS system with high quality output. The system is composed of three separate neural network blocks: prosody prediction, acoustic feature prediction and Linear Prediction Coding Net as a neural vocoder. This system can synthesize speech with close to natural quality while running 3 times faster than real-time on a standard CPU. The modular setup of the system allows for simple adaptation to new voices with a small amount of data. We first demonstrate the ability of the system to produce high quality speech when trained on large, high quality datasets. Following that, we demonstrate its adaptability by mimicking unseen voices using 5 to 20 minutes long datasets with lower recording quality. Large scale Mean Opinion Score quality and similarity tests are presented, showing that the system can adapt to unseen voices with quality gap of 0.12 and similarity gap of 3% compared to natural speech for male voices and quality gap of 0.35 and similarity of gap of 9 % for female voices.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Sheng Zhao
7. Yanqing Liu
8. Yi Ren

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Sheng Zhao
7. Ryuichi Yamamoto
8. Yanqing Liu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Efficient Adaptation Few-Data Adaptation [43, 9, 173, 234, 439, 48, 39, 230] Untranscribed Data Adaptation [397, 130, 215] Few-Parameter Adaptation [9, 43, 39] Zero-Shot Adaptation [9, 43, 139, 55]”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “They fine-tune the whole model (Chen et al., 2018; Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters.”
• Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, background): “An effective approach for speaker adaptation in neural TTS is to fine-tune all or part of model with a small amount of data from the target speaker [5, 6].”
• Fastpitch: Parallel Text-to-Speech with Pitch Prediction (2020, methodology): “In particular, fundamental frequency modeled with a neural network has been repeatedly shown to improve the quality of synthesized speech in a concatena-tive model [6], and later fully neural models [11, 12].”
• The AS-NU System for the M2VoC Challenge (2021, background): “Such a challenging task is referred to as voice cloning, speaker adaptation or few-shot/one-shot speech synthesis [7, 8, 9, 10, 11, 12].”
• Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis (2021, background): “Some works focused on solving the problem of noisy data [7, 9, 19, 20], e.”
• Improving LPCNET-Based Text-to-Speech with Linear Prediction-Structured Mixture Density Network (2020, methodology): “To further improve the performance of speech synthesis system, a recently proposed LPCNet vocoder takes advantages of the merits of both the source-filter model-based parametric vocoder and the WaveRNN structure [5, 6].”
• Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS? (2020, unknown): “The first approach is simple fine-tuning [5, 6, 7]: the TTS model receives a small amount”
• Gaussian Lpcnet for Multisample Speech Synthesis (2020, background): “It has become quite popular [16] due to high quality of the synthesized speech and efficiency both in terms of time and memory.”
• Lightweight LPCNet-Based Neural Vocoder with Tensor Decomposition (2020, methodology): “The obvious superiority of Original LPCNet over STRAIGHT confirms that LPCNet is also useful as a vocoder for TTS, as reported in [26].”
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding (2020, unknown): “Previous studies led to decent results for few-shot TTS, which, however, require an additional fine-tuning process [6, 7, 8, 9, 11].”
• Investigation of Fast and Efficient Methods for Multi-Speaker Modeling and Speaker Adaptation (2021, methodology): “The first approach pre-tains multi-speaker TTS models on a large multi-speaker dataset and then fine-tunes the models on a small dataset of target speaker [7, 8].”
• The STC text-to-speech system for Blizzard Challenge 2019 (2019, background): “It is also not the first published usage of LPCNet in text-to-speech systems [34].”
• Bunched LPCNet : Vocoder for Low-cost Neural Text-To-Speech Systems (2020, background): “It is one of the most compute efficient neural vocoders in recent times and has been demonstrated to give excellent performance when used in text-to-speech systems [4, 5] as well as in speech codecs [6].”