Gated Feedback Recurrent Neural Networks

By Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio. 2015

897 citations recorded by the source corpus.

In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GFRNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “Stacked LSTM (Graves, 2013) 1.67 GF-LSTM (Chung et al., 2015) 1.58 Grid-LSTM (Kalchbrenner et al., 2016a) 1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm…”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “, we also evaluate our model on enwik8(Mahoney 2009) which is 100M bytes of unprocessed Wikipedia text, including markup and non-Latin characters. There are 205 unique bytes in the dataset. Following Chung et al. (2015), and as in text8, we split the data into 90M, 5M and 5M for training, dev and te”
An efficient framework for learning sentence representations (2018, methodology): “Weights of the GRU are initialized using uniform Xavier initialization and gate biases are initialized to 1.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “Finally, our stack representation is related to the multiplicative gate mechanism used in LSTM and the Gated Recurrent Unit (GRU, Cho et al. (2014); Chung et al. (2015)).”
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “BERT+BUTD Bottom-Up and Top-Down (BUTD) attention (Anderson et al., 2018) method encodes questions with GRU (Chung et al., 2015), then attends to object RoI features {fj} to predict the answer.”
HyperNetworks (2016, background): “…already does a formidable job of generating time-varying parameters of a Mixture Gaussian distribution used to generate realistic handwriting samples, the ability to go one level deeper, and to dynamically generate the generative model is one of the key advantages of HyperRNN over a normal RNN.”
Recurrent Highway Networks (2016, background): “This description of the problem in terms of largest singular values or the spectral radius sheds light on boundary conditions for vanishing and exploding gradients yet does not illuminate how the eigenvalues are distributed overall.”
Grid Long Short-Term Memory (2015, methodology): “We follow the splitting procedure of [4], where the last 5 million characters are used for testing.”
End-to-end attention-based large vocabulary speech recognition (2015, methodology): “In the spirit of he Clockwork RNN (Koutnik et al., 2014) and hierarchical gating RNN (Chung et al., 2015), we introduce a recurrent architecture that successively reduces source sequence length by pooling frames neighboring in time.”
Hierarchical Multiscale Recurrent Neural Networks (2016, background): “, 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, methodology): “ecurrent neural network with gated recurrent units (GRU). We call this decoder a base decoder. Second, we build a novel two-layer recurrent neural network, inspired by the gated-feedback network from Chung et al. (2015), called a biscale recurrent neural network. We design this network to facilitate”
On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “Recently there has been a resurgence of new structural designs for recurrent neural networks (RNNs) [1, 2, 3].”
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification (2015, methodology): “, 2015) and gated neural network (Cho et al., 2014; Chung et al., 2015).”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, methodology): “For CIFAR-100 we use VGG-16 (Simonyan & Zisserman, 2014) with an additional 512x100 linear layer added at the end to make the ﬁnal representations 100 dimensional, while for Wiki-3029 we use a Gated Recurrent Network (GRU) (Chung et al., 2015) with output dimension 300 and ﬁx the word embedding laye”
Temporal Convolutional Networks for Action Segmentation and Detection (2016, methodology): “For many of these models, such as RNNs with LSTM or GRUs [5], the latent state at each time step, t, is only a function of the data at t and the hidden state and memory at t − 1.”
Multiplicative LSTM for sequence modelling (2016, neutral): “…were left in their raw form, containing punctuation and both architecture test error (bits/char) 7-layer stacked LSTM (Graves, 2013) 1.67 gf-LSTM (Chung et al., 2015) 1.58 2-layer stacked LSTM (ours) 1.53 grid LSTM (Kalchbrenner et al., 2015) 1.48 multiplicative integration LSTM (Wu et al.,…”