Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices

By Heiga Zen, Yannis Agiomyrgiannakis, Niels Egberts, Fergus Henderson, Przemyslaw Szczepaniak. 2016

122 citations recorded by the source corpus.

Acoustic models based on long short-term memory recurrent neural networks (LSTM-RNNs) were applied to statistical parametric speech synthesis (SPSS) and showed significant improvements in naturalness and latency over those based on hidden Markov models (HMMs). This paper describes further optimizations of LSTM-RNN-based SPSS for deployment on mobile devices; weight quantization, multi-frame inference, and robust inference using an {\epsilon}-contaminated Gaussian loss function. Experimental results in subjective listening tests show that these optimizations can make LSTM-RNN-based SPSS comparable to HMM-based SPSS in runtime speed while maintaining naturalness. Evaluations between LSTM-RNN- based SPSS and HMM-driven unit selection speech synthesis are also presented.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yu Zhang
5. Yoshua Bengio
6. Junichi Yamagishi
7. Quoc V. Le
8. Frank K. Soong

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Yoshua Bengio
4. Quoc V. Le
5. Yu Zhang
6. Xu Tan
7. Kyunghyun Cho
8. Dong Yu

In Speech & Audio

1. Xu Tan
2. Junichi Yamagishi
3. Yu Zhang
4. Tao Qin
5. Frank K. Soong
6. Tie-Yan Liu
7. Lei He
8. Ron J. Weiss

In Signal Processing

1. Junichi Yamagishi
2. Xu Tan
3. Yu Zhang
4. Frank K. Soong
5. Tao Qin
6. Tie-Yan Liu
7. Heiga Zen
8. Dong Yu

Citations in the corpus

• WaveNet: A Generative Model for Raw Audio (2016, methodology): “Zen et al. (2016) showed that state-of-the-art statistical parametric speech syntheziers matched state-of-the-art concatenative ones in some languages.”
• Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions (2017, neutral): “We also compare to the original Tacotron that predicts linear spectrograms and uses Griffin-Lim to synthesize audio, as well as concatenative [30] and parametric [31] baseline systems, both of which have been used in production at Google.”
• Tacotron: Towards End-to-End Speech Synthesis (2017, methodology): “We compare our model with a parametric (based on LSTM (Zen et al., 2016)) and a concatenative system (Gonzalvo et al., 2016), both of which are in production.”
• A Survey on Neural Speech Synthesis (2021, background): “Speedup with Domain Knowledge Domain knowledge from speech can be leveraged to speed up inference, such as linear prediction [357], multiband modeling [411, 400], subscale prediction [147], multi-frame prediction [420, 376, 367, 123], streaming synthesis [74], etc.”
• Sample Efficient Adaptive Text-to-Speech (2018, methodology): “To achieve this, we normalize the fundamental frequency values to have zero mean and unit variance separately for each speaker during training, denoted as ˆ f 0 : As mentioned earlier, at test time, we use an existing model (Zen et al., 2016) to predict ( l , ˆ f 0 ) .”
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018, background): “The past few years have seen exciting developments in the use of deep neural networks to synthesize natural-sounding human speech (Zen et al., 2016; van den Oord et al., 2016; Wang et al., 2017a; Arik et al., 2017; Taigman et al., 2017; Shen et al., 2017).”
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “With regards to single-speaker speech synthesis, deep learning has been used for a variety of subcomponents, including duration prediction (Zen et al., 2016), fundamental frequency prediction (Ronanki et al., 2016), acoustic modeling (Zen and Sak, 2015), and more recently autoregressive…”
• Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model (2020, methodology): “The input is usually enriched by information about syntax, using taggers and parsers [3, 4, 5].”
• DurIAN: Duration Informed Attention Network For Multimodal Synthesis (2019, methodology): “In DurIAN, we replace the attention mechanism with an alignment model [15, 16], in which the alignment between the phoneme sequence and the target acoustic sequence is inferred from a phoneme duration prediction model.”
• On Using Backpropagation for Speech Texture Generation and Voice Conversion (2017, methodology): “Ad-ditionally, neural networks can be used to generate new data, having been applied to speech synthesis [7, 8], image generation [9], and image inpainting and superresolution [10].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “For example, TalkNet [12] uses the CTC-based automatic speech recognition (ASR) model Quartznet [111], FastSpeech 2 [174] uses the forcedalignment tool MFA toolkit [135], DurIAN [243] uses a external alignment model [51, 252], RobuTrans [119] uses speech recognition tools, Parallel Tacotron [49] and”
• Fcl-Taco2: Towards Fast, Controllable and Lightweight Text-to-Speech Synthesis (2021, background): “Previous work mostly focused on the optimization of vocoder for edge deployment [3-5], while less attention has been paid to the acoustic model [6].”
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network (2019, methodology): “The zero vector can be chosen, which, by design, is the mean of the sentence prosody embedding space, and in our experience tends to represent prosody which exhibits the averaging effect as seen in models such as (Zen et al., 2016).”
• EMPHASIS: An Emotional Phoneme-based Acoustic Model for Speech Synthesis System (2018, methodology): “The Bi-LSTM system produced fuzzy speech due to the over-smoothness of the acoustic parameters, while this phenomenon was alleviated in our system.”
• Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks (2017, methodology): “In contrast, our algorithm can directly utilize original synthesis procedures [43].”
• DurIAN: Duration Informed Attention Network for Speech Synthesis (2020, methodology): “In DurIAN, we replace the attention mechanism with an alignment model [20, 21], in which the alignment between the phoneme sequence and the target acoustic sequence is inferred from a phoneme duration prediction model as illustrated in Figure.”
• Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “[431] Weight quantization LSTM Falcao et al.”
• TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model (2020, methodology): “Zen et al [8, 9] proposed a hybrid NN–parametric TTS model, where deep neural networks are used to predict the phoneme duration and to generate frame-level acoustic features.”