Investigating gated recurrent neural networks for speech synthesis

By Zhizheng Wu, Simon King. 2016

98 citations recorded by the source corpus.

Recently, recurrent neural networks (RNNs) as powerful sequence models have re-emerged as a potential acoustic model for statistical parametric speech synthesis (SPSS). The long short-term memory (LSTM) architecture is particularly attractive because it addresses the vanishing gradient problem in standard RNNs, making them easier to train. Although recent studies have demonstrated that LSTMs can achieve significantly better performance on SPSS than deep feed-forward neural networks, little is known about why. Here we attempt to answer two questions: a) why do LSTMs work well as a sequence model for SPSS; b) which component (e.g., input gate, output gate, forget gate) is most important. We present a visual analysis alongside a series of experiments, resulting in a proposal for a simplified architecture. The simplified architecture has significantly fewer parameters than an LSTM, thus reducing generation complexity considerably without degrading quality.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “Others ensured syntactic or semantic coherence in different ways, such as filtering replacements by word similarity or sentence similarity (Alzantot et al., 2018; Kuleshov et al., 2018), or by using synonyms and other word lists (Samanta and Mehta, 2017; Yang et al., 2018).”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, background): “tain quantitative correlations between parts of the neural network and linguistic properties, in both speech (Wu and King, 2016; Alishahi et al., 2017; Belinkov and Glass, 2017; Wang et al., 2017) and language processing models (Köhn, 2015; Qian et al.”
Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices (2016, methodology): “They were successfully applied to acoustic modeling for SPSS [8–11].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “Others have correlated activations of gated recurrent networks with phoneme boundaries in autoencoders [15] and in a text-to-speech system [16].”
Unsupervised Learning for Expressive Speech Synthesis (2017, methodology): “Other types of NN-based architectures were proposed, for instance Wu et al. (2015b) apply a bottleneck DNN design, i.e. a hidden layer with a relatively smaller amount of neurons than the prior and the posterior layers, for a compact representation, achieving a better modeling of frame interdependen”
Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention (2017, unknown): “[9, 10, 11, 12], and recently proposed techniques e.”
Gate Activation Signal Analysis for Gated Recurrent Neural Networks and its Correlation with Phoneme Boundaries (2017, result): “This is consistent to the previous finding that the sudden bursts of gate activations indicated that there were boundaries of phonemes in a speech synthesis task [23].”
Siri On-Device Deep Learning-Guided Unit Selection Text-to-Speech System (2017, methodology): “The recurrent layer employs gated recurrent units (GRU) [12], which is an alternative implementation of the long short-term memory (LSTM), but uses fewer model parameters to achieve similar performance, as validated in statistical parametric speech synthesis [13].”
Forward–Backward Decoding Sequence for Regularizing End-to-End TTS (2019, background): “In direct contrast to concatenative synthesis, statistical parametric speech synthesis (SPSS) [11] has also grown in popularity over the last years [12]–[16].”
Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis (2020, background): “Deep neural network (DNN)-based approaches have become mainstream in statistical parametric text-to-speech (TTS) synthesis in recent years [1, 2, 3].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “This may be computed directly, for instance by computing the correlation between long short-term memory (LSTM) cell activations and Mel-frequency cepstral coefficient (MFCC) acoustic features [345], or indirectly, by defining discrimination tasks based on activations [5, 57].”
Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, methodology): “Merlin is designed for DNN-based SPSS systems and it supports various types of neural networks including mixture density networks (MDN) [18] and recurrent neural networks (RNNs) [19], [20].”
EMPHASIS: An Emotional Phoneme-based Acoustic Model for Speech Synthesis System (2018, background): “D convolution banks, highway layers and bi-directional GRU layers (CBHG) [9], [13].”
Minimal gated unit for recurrent neural networks (2016, methodology): “When we have accomplished this work we noticed a paper accepted into the ICCASP conference by Wu and King ([27]).”
Merlin: An Open Source Neural Network Speech Synthesis System (2016, methodology): “In our implementation, the several variants described in [30] are also available.”