Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
By Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio. 2014
14,897 citations recorded by the source corpus.
In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM.
Subfields: Artificial Intelligence, Model Architectures, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Sergey Levine
- Kyunghyun Cho
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Richard Socher
- Ruslan Salakhutdinov
- Yoav Goldberg
- Mohit Bansal
- Jianfeng Gao
In Signal Processing
- Yoshua Bengio
- Jinyu Li
- Dong Yu
- Hung-yi Lee
- Shinji Watanabe
- Junichi Yamagishi
- Yu Zhang
- James R. Glass
Citations in the corpus
- Attention is All you Need (2017, background): “Recurrent neural networks, long short-term memory [12] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [29, 2, 5].”
- A Structured Self-attentive Sentence Embedding (2017, methodology): “Several models have been proposed along this line, by using recurrent networks (Hochreiter & Schmidhuber, 1997; Chung et al., 2014), recursive networks (Socher et al.”
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “GRUs have been shown to perform just as well as LSTM on several sequence prediction tasks [4] while being simpler.”
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “Both RNN enc and RNN dec were single-layer GRUs (Chung et al., 2014) with h dim = 512 , while RNN enc was bidirectional and RNN dec was unidirectional.”
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “To alleviate the difficulty of training RNNs due to vanishing and exploding gradients (Bengio et al., 1994), we use Gated Recurrent Units (GRUs) (Cho et al., 2014) which have been demonstrated to have good empirical performance (Chung et al., 2014; J´ozefowicz et al., 2015).”
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “This is the same classification strategy adopted by Conneau et al. (2017). We train on a collection of about 1 million sentence pairs from the SNLI (Bowman et al.”
- Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “The encoder consists of a bidirectional GRU-RNN (Chung et al., 2014), while the decoder consists of a uni-directional GRU-RNN with the same hidden-state size as that of 1 http://duc.nist.gov/ the encoder, and an attention mechanism over the source-hidden states and a soft-max layer over target…”
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, methodology): “, 2014) has become a dominant framework using encoder-decoder architectures based on RNNs (Chung et al., 2014; Hochreiter & Schmidhuber, 1997) and more recently Transformers (Vaswani et al.”
- Improving Neural Language Models with a Continuous Cache (2016, methodology): “Several architecture for recurrent networks have been proposed, such as the Elman network (Elman, 1990), the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) or the gated recurrent unit (GRU) (Chung et al., 2014).”
- Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Given feature representation for words in a sentence, the sentence embedding s is found from the hidden state of the GRU.”
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “These models are inspired by sequence-to-sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) but use CNN-based encodings instead of RNNs (Hochreiter and Schmidhuber, 1997; Chung et al., 2014).”
- Efficient Neural Audio Synthesis (2018, methodology): “The matrix R formed from the matrices R u , R r , R e is computed as a single matrix-vector product to produce the contributions to all three gates u t , r t and e t (a variant of the GRU cell as in (Chung et al., 2014; Engel, 2016).) σ and τ are the standard sigmoid and tanh non-linearities.”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “A.1 Visual Question Answering Model Architecture Pythia encodes the question with an attention-based GRU (Chung et al., 2014) network and fuse the information with a multi-modal factorized bilinear pooling network.”
- Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “− → h i is obtained by recursively computing f(xi, − → h i−1) where f is a Gated Recurrent Unit (GRU) (Chung et al., 2014), and ←− h i is computed analogously.”
- Learning to Extract Coherent Summary via Deep Reinforcement Learning (2018, background): “Gated recurrent unit is a variant of recurrent neural network proposed by (Chung et al. 2014).”