Learning Longer Memory in Recurrent Neural Networks

By Tomas Mikolov, Armand Joulin, Sumit Chopra, Michaël Mathieu, Marc'Aurelio Ranzato. 2014

269 citations recorded by the source corpus.

Recurrent neural network is a powerful model that learns temporal patterns in sequential data. For a long time, it was believed that recurrent networks are difficult to train using simple optimizers, such as stochastic gradient descent, due to the so-called vanishing gradient problem. In this paper, we show that learning longer term patterns in real data, such as in natural language, is perfectly possible using gradient descent. This is achieved by using a slight structural modification of the simple recurrent neural network architecture. We encourage some of the hidden units to change their state slowly by making part of the recurrent weight matrix close to identity, thus forming kind of a longer term memory. We evaluate our model in language modeling experiments, where we obtain similar performance to the much more complex Long Short Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997).

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “…is different when training the small NNLM model on either the Penn Treebank (Marcus et al., 1993) or text8 datasets (for PTB, we used the same train/validation/test set split and vocabulary as (Mikolov et al., 2011), while on text8 we used the split/vocabulary from (Mikolov et al., 2014)).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014); Koutnik et al. (2014); Wu et al. (2016); Li et al. (2018).”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “…able to keep information from the broader context into memory, and possibly even perform simple forms of reasoning about it (Hermann et al., 2015; Hochreiter and Schmidhuber, 1997; Ji et al., 2015; Mikolov et al., 2015; Sordoni et al., 2015; Sukhbaatar et al., 2015; Wang and Cho, 2015, a.o.).”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, background): “Our definition of a stack in a recurrent net is through constraining part of the recurrent matrix, similar to (Mikolov et al., 2014) where it was shown that diagonal recurrent matrix can help the recurrent net to store longer memory.”
Improving Neural Language Models with a Continuous Cache (2016, background): “Model Test LSTM-500 (Mikolov et al., 2014) 156 SCRNN (Mikolov et al., 2014) 161 MemNN (Sukhbaatar et al., 2015) 147”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
Efficient softmax approximation for GPUs (2016, methodology): “When the vocabulary is large, this step is computationally expensive and often dominates the computation of the whole model (Joze-fowicz et al., 2016; Mikolov et al., 2014), as discussed in introduction and related work.”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, methodology): “For natural language, the Recurrent Neural Network (RNN [7, 27]) and the Long Short-Term Memory network (LSTM [12]) are also widely used in machine translation [13, 5, 35] and speech recognition [28].”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “For the word-level task, we use similar settings to Semeniuta et al. [2016] and Mikolov et al. [2014].”
Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “in effective training algorithms of deep networks including: • Using Rectified Linear Units (ReLUs) as activation function [27], • Introducing dropout methods [28], • Random initialization for the weights of the network [29], • Addressing the degradation of training accuracy by residual learning net”
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015, methodology): “Our initialization scheme bears some resemblance to the idea of Mikolov et al. [26], where a part of the weight matrix is ﬁxed to identity or approximate identity.”
Dynamic Evaluation of Neural Sequence Models (2017, methodology): “We use the same test set as in Mikolov et al. (2014), but also hold out the final 100k training tokens as a validation set to allow for fair hyper-parameter tuning (the original task did not have a validation set).”
Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2015, background): “However, they lack a high-level and intuitive spatio-temporal structure though they have been shown to be successful at modeling long sequences [49, 43, 52].”
Weakly Supervised Memory Networks (2015, background): “In practice, the performance gains over carefully trained RNNs are modest (Mikolov et al., 2014).”
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images (2015, methodology): “For natural language, the Recurrent Neural Network (RNN [13, 36]) and the Long-Short Term Memory (LSTM [20]) achieve the state-of-the-art performance in many NLP tasks such as machine translation [21, 8, 47] and speech recognition [37].”
Larger-Context Language Modelling with Recurrent Neural Network (2015, background): “More recently, Mikolov et al. (2014) proposed a similar approach however without relying on external topic modelling.”
Resnet in Resnet: Generalizing Residual Architectures (2016, background): “The SCRN architecture of Mikolov et al. (2014) also uses hidden and context units together within a single layer to learn longer term information, which behave similarly to the transient and residual streams, but SCRN only allows unidirectional ﬂow from context to hidden units and connections…”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “We depart from the notation in (Mikolov et al., 2014) and reuse the symbols used in the LSTM description.”