TTS synthesis with bidirectional LSTM based recurrent neural networks

By Yuchen Fan, Yao Qian, Fenglong Xie, Frank K. Soong. 2014

501 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Junichi Yamagishi
2. Xu Tan
3. Tao Qin
4. Tie-Yan Liu
5. Frank K. Soong
6. Dong Yu
7. Yoshua Bengio
8. Jinyu Li

In Natural Language Processing

1. Junichi Yamagishi
2. Dong Yu
3. Zhenhua Ling
4. Yoshua Bengio
5. Tie-Yan Liu
6. Tao Qin
7. Li Deng
8. Kyunghyun Cho

In Speech & Audio

1. Junichi Yamagishi
2. Xu Tan
3. Lei He
4. Frank K. Soong
5. Tao Qin
6. Sheng Zhao
7. Tie-Yan Liu
8. Tomoki Toda

Citations in the corpus

• WaveNet: A Generative Model for Raw Audio (2016, background): “…(Yoshimura, 2002), feed-forward neural networks (Zen et al., 2013), and recurrent neural networks (Tuerk & Robinson, 1993; Karaali et al., 1997; Fan et al., 2014), is trained from the extracted vocoder parameters and linguistic features as where Λ denotes the set of parameters of the…”
• LSTM: A Search Space Odyssey (2015, methodology): “…language modeling (Zaremba et al., 2014) and translation (Luong et al., 2014), acoustic modeling of speech (Sak et al., 2014), speech synthesis (Fan et al., 2014), protein secondary structure prediction (Sønderby & Winther, 2014), analysis of audio (Marchi et al., 2014), and video data…”
• Deep learning in neural networks: An overview (2014, background): “…(GonzalezDominguez et al., 2014), medium-vocabulary speech recognition (Geiger et al., 2014), prosody contour prediction (Fernandez et al., 2014), audio onset detection (Marchi et al., 2014), text-to-speech synthesis (Fan et al., 2014), and social signal classification (Brueckner & Schulter, 2014).”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “With the advancement of modern neural networks, some work initially replaced HMMs with recurrent neural networks (RNNs) but still followed the SPSS paradigm (Fan et al., 2014; Zen and Sak, 2015; Valentini-Botinhao et al., 2016).”
• A Survey on Neural Speech Synthesis (2021, methodology): “Further, WaveRNN [441] is developed for efficient audio synthesis, using a recurrent neural network and leveraging several designs including dual softmax layer, weight pruning, and subscaling techniques to reduce the computation.”
• Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices (2016, methodology): “They were successfully applied to acoustic modeling for SPSS [8–11].”
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “Many recent state-of-the-art results in sequence processing are based on LSTM, which learned to control robots [159], and was used to set benchmark records in prosody contour prediction [55] (IBM), textto-speech synthesis [54] (Microsoft), large vocabulary speech recognition [213] (Google), and mach”
• Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis (2019, background): “quency (F0) [5], [6], cepstral or spectral features [7], [8], [9], and duration [10], [11].”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Rich unit selection based concatenative speech synthesis [8] and statistical parametric speech synthesis like HMM/DNN/LSTM based models [9, 10, 11], have been the most popular methods in the past.”
• Deep Feed-Forward Sequential Memory Networks for Speech Synthesis (2018, methodology): “thesis back-end modeling[1, 2]. Compared with conventional HMM-GMM methods training at state level, neural network based methods model and predict at much smaller step, e.g. frame or even sample level[2, 3, 4]. Recurrent Neural Networks (RNNs) have shown their advantage in modeling long-term depende”
• Generative Adversarial Network-Based Glottal Waveform Model for Statistical Parametric Speech Synthesis (2017, background): “Recent use of neural network-based acoustic models [3], especially sequence models, such as long short-term memory (LSTM) networks [4, 5], have addressed primarily the acoustic modelling accuracy and to some extent also the over-smoothing problem [6, 7].”
• Deep Recurrent Neural Networks for Human Activity Recognition (2017, methodology): “Bidirectional DRNNs have been used in many domains, such as speech recognition and text-to-speech synthesis [1,14].”
• Investigating gated recurrent neural networks for speech synthesis (2016, unknown): “In [18], a bidirectional LSTM was employed to map a sequence of linguistic features to the corresponding sequence of acoustic features.”
• On the localness modeling for the self-attention based end-to-end speech synthesis (2020, unknown): “…of neural network research in recent years, parametric speech synthesis advances from hidden Markov models (HMM) to neural networks (NN) (Black, Zen, & Tokuda, 2007; Fan, Qian, Xie, & Soong, 2014; Ling et al., 2015; Watts, Henter, Merritt, Wu, & King, 2016; Ze, Senior, & Schuster, 2013).”
• Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis (2018, unknown): “In the text-to-speech systems that use vocoders, extracted acoustic features are modeled by HMM-based or DNN-based acoustic models that represent the relationship between linguistic and speech features [1, 3, 9, 10].”
• Adapting and controlling DNN-based speech synthesis using input codes (2017, background): “As future work, it would be interesting to consider similar input code setups for BLSTM-based acoustic models [29, 24].”
• Merlin: An Open Source Neural Network Speech Synthesis System (2016, methodology): “To include contextual constraints, a bidirectional long short-term memory (LSTM) based recurrent neural network (RNN) was employed in [27] to formulate TTS as a sequence to sequence mapping problem, that is to map a sequence of linguistic features to the corresponding sequence of acoustic features.”
• Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks (2016, background): “LSTMs have been applied in a range of speech problems [15, 16], including regression problems such as text-to-speech [17, 18, 19, 20, 21] and as previously mentioned speech enhancement [10, 8].”
• GELP: GAN-Excited Liner Prediction for Speech Synthesis from Mel-spectrogram (2019, background): “Introduction of deep neural networks for acoustic modeling [1] in statistical parametric speech synthesis (SPSS) [2] started the ongoing trend of deep learning in TTS, and was soon followed by recurrent networks [3], and more recently, sequence-to-sequence models with attention [4, 5].”