Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis

By Heiga Zen, Hasim Sak. 2015

320 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Lei He
7. Heiga Zen
8. Tomoki Toda

In Speech & Audio

1. Xu Tan
2. Junichi Yamagishi
3. Tao Qin
4. Tie-Yan Liu
5. Frank K. Soong
6. Tomoki Toda
7. Lei He
8. Sheng Zhao

In Signal Processing

1. Junichi Yamagishi
2. Xu Tan
3. Frank K. Soong
4. Tao Qin
5. Tie-Yan Liu
6. Lei He
7. Tomoki Toda
8. Haizhou Li

Citations in the corpus

• Deep Voice: Real-time Neural Text-to-Speech (2017, background): “…TTS system components, including grapheme-to-phoneme conversion models (Rao et al., 2015; Yao & Zweig, 2015), phoneme duration prediction models (Zen & Sak, 2015), fundamental frequency prediction models (Pascual & Bonafonte, 2016; Ronanki et al., 2016), and audio synthesis models (van den Oord…”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “With the advancement of modern neural networks, some work initially replaced HMMs with recurrent neural networks (RNNs) but still followed the SPSS paradigm (Fan et al., 2014; Zen and Sak, 2015; Valentini-Botinhao et al., 2016).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Further, WaveRNN [441] is developed for efficient audio synthesis, using a recurrent neural network and leveraging several designs including dual softmax layer, weight pruning, and subscaling techniques to reduce the computation.”
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “…a variety of subcomponents, including duration prediction (Zen et al., 2016), fundamental frequency prediction (Ronanki et al., 2016), acoustic modeling (Zen and Sak, 2015), and more recently autoregressive sample-by-sample audio waveform generation (e.g., Oord et al., 2016; Mehri et al., 2016).”
• Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices (2016, methodology): “The architecture of the acoustic LSTM-RNNs was 1 × 128-unit ReLU [38] layer followed by 3 × 128-cell LSTMP layers [39] with 64 recurrent projection units with a linear recurrent output layer [9].”
• Voice Morphing that improves TTS quality using an optimal dynamic frequency warping-and-weighting transform (2016, background): “Recent developments in TTS allow us to significantly improve the quality of the synthesized speech, namely by using an LSTM to predict the acoustic parameters from the linguistic ones [10] and Vocaine as the vocoder [2].”
• Merlin: An Open Source Neural Network Speech Synthesis System (2016, background): “In [28], LSTM with a recurrent output layer was proposed to include contextual constraints.”
• Postfilters to Modify the Modulation Spectrum for Statistical Parametric Speech Synthesis (2016, methodology): “This utterance-level post-filter is further extended to a segment-level post-filter to modify the MS segment by segment in order to achieve low-delay parameter generation [42], [43].”
• Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech (2016, unknown): “LSTMs have been applied in a range of speech problems [17, 18], including regression problems such as text-to-speech [19, 20, 21, 22, 23] and as previously mentioned speech enhancement [8, 9].”
• Tacotron-Based Acoustic Model Using Phoneme Alignment for Practical Neural Text-to-Speech Systems (2019, methodology): “In CAM, frame-level full-context label vectors were obtained from phoneme-level vectors with three numerical features for the coarse-coded position of the current frame in the current phoneme and one numerical feature for the duration of the current segment, as in [4](3).”
• Sequence-to-sequence Modelling of F0 for Speech Emotion Conversion (2019, methodology): “In particular LSTM-RNN networks have been used to model timbre and prosody variations over time, improving the speech quality over models [13] in text-to-speech synthesis [14, 15] and voice conversion [16, 17].”
• Graphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis (2020, background): “In the recent past, concatenative [4] and statistical parametric speech synthesis [5–7] systems were the mainstream techniques.”
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network (2019, background): “This trend continued with the move to deep neural networks (Zen et al., 2013; Zen & Sak, 2015).”
• TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model (2020, methodology): “Zen et al [8, 9] proposed a hybrid NN–parametric TTS model, where deep neural networks are used to predict the phoneme duration and to generate frame-level acoustic features.”
• Statistical parametric speech synthesis: from HMM to LSTM-RNN (2015, methodology): “Advantages of RNN-based acoustic modeling for SPSS • Model dependency between frames − HMM: discontinuous (step-wise) → smoothing − DNN: discontinuous (frame-by-frame mapping) [35] → smoothing − RNN: smooth [36, 35] • Low latency − Unidirectional structure allows fully frame-level streaming [35] • M”