How to Construct Deep Recurrent Neural Networks

By Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio. 2013

1,083 citations recorded by the source corpus.

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Signal Processing

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “The compared models are: KN 5-gram (Mikolov et al., 2011), RNN (Mikolov et al., 2011), LSTM (Graves, 2013), Stack / Deep RNN (Pascanu et al., 2013), FOFE-FNN (Zhang et al., 2015), Noisy LSTM (G¨ulc¸ehre et al., 2016), and the small model from (Zaremba et al., 2014).”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “In both cases, we use a multilayer network with a single maxout (Goodfellow et al. , 2013) hidden layer to compute the conditional probability of each target word (Pascanu et al. , 2014).”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “The computation from the hidden state in the decoder to the output is implemented as a deep neural network (Pascanu et al., 2014) with a single intermediate layer having 500 maxout units each pooling 2 inputs (Goodfellow et al.”
Pointer Sentinel Mixture Models (2016, background): “The gradients are rescaled if their global norm exceeds 1 (Pascanu et al., 2013b).3 We evaluate the medium model configuration which features a hidden size of H = 650 and a two layer LSTM.”
Learning to Execute (2014, methodology): “From a methodological perspective, we formulate the program evaluation task as a language modeling problem on a sequence (Mikolov, 2012; Sutskever, 2013; Pascanu et al., 2013).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “We set the momentum to 0.99 and renormalize gradients if the norm exceeds 0.1 (Pascanu et al., 2014).”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “In this work, we use a deep output layer (Pascanu et al., 2014) to compute the output word probability.”
Recurrent Neural Network Regularization (2014, background): “Language modeling is the ﬁrst task where RNNs have achieved substantial success (Mikolov et al., 2010; 2011; Pascanu et al., 2013).”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “We also use residual connections across each convolutional layer and apply weight normalization (Salimans and Kingma, 2016) to speed up convergence.”
Character-Aware Neural Language Models (2015, background): “Indeed, having multiple layers is often crucial for obtaining competitive performance on various tasks (Pascanu et al. 2013).”
On Using Monolingual Corpora in Neural Machine Translation (2015, methodology): “We use a deep output layer (Pascanu et al. , 2014) to compute the conditional distribution over words: where y t is an one-hot encoded vector indicating one of the words in the target vocabulary.”
Recurrent Highway Networks (2016, methodology): “We compare optimization of RHNs to DT-RNNs and DT(S)RNNs (Pascanu et al., 2013).”
Pointing the Unknown Words (2016, methodology): “We use a deep output layer (Pascanu et al., 2013) to compute the conditional distribution over words: where W is a learned weight matrix and b is a bias of the output layer. f o is a single-layer feed-forward neural network. ψ ( W o , b o ) ( · ) is a function that performs an afﬁne transformation…”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “The learning rate is set to 0.001 and the gradients are clipped to a maximum norm of 1.0 [Pascanu et al., 2013].”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “The norm of the gradient is clipped with a threshold 1 (Pascanu et al., 2013).”