Grid Long Short-Term Memory

By Nal Kalchbrenner, Ivo Danihelka, Alex Graves. 2015

373 citations recorded by the source corpus.

This paper introduces Grid Long Short-Term Memory, a network of LSTM cells arranged in a multidimensional grid that can be applied to vectors, sequences or higher dimensional data such as images. The network differs from existing deep LSTM architectures in that the cells are connected between network layers as well as along the spatiotemporal dimensions of the data. The network provides a unified way of using LSTM for both deep and sequential computation. We apply the model to algorithmic tasks such as 15-digit integer addition and sequence memorization, where it is able to significantly outperform the standard LSTM. We then give results for two empirical tasks. We find that 2D Grid LSTM achieves 1.47 bits per character on the Wikipedia character prediction benchmark, which is state-of-the-art among neural approaches. In addition, we use the Grid LSTM to define a novel two-dimensional translation model, the Reencoder, and show that it outperforms a phrase-based reference system on a Chinese-to-English translation task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “…have been proposed for modelling translation, such as encoder-decoder networks (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Kaiser & Bengio, 2016), networks with attentional pooling (Bahdanau et al., 2014) and two-dimensional networks (Kalchbrenner et al., 2016a).”
Can Active Memory Replace Attention? (2016, methodology): “The experiments from [22] are only performed on a very small dataset of 44K short sentences.”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “They decreased the number of nonlinear activations and path length, but the gradient propagation still relies on the recurrent computation.”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “This tighter integration is potentially helpful for sequence to sequence transduction tasks which involve a significant amount of reordering, and is similar in spirit to the Grid LSTM Reencoder (Kalchbrenner et al., 2016).”
Training Very Deep Networks (2015, methodology): “More recently, a similar LSTMinspired model was also proposed [32].”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “make every network very deep and design its architecture in such a way as to mitigate the vanishing gradient problem [13] associated with long chains of iteration [29, 17].”
Pixel Recurrent Neural Networks (2016, methodology): “In subsequent layers, the channels are also connected to themselves. previous approaches that use gating along the depth of the recurrent network (Kalchbrenner et al., 2015; Zhang et al., 2016), but has the advantage of not requiring additional gates.”
HyperNetworks (2016, positive): “…RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM (Kalchbrenner et al., 2016) 1.47 16.8 M LSTM (Rocki, 2016b) 1.45 MI-LSTM (Wu et al., 2016) 1.44 Recurrent Highway Networks (Zilly et al., 2016) 1.42…”
Video Pixel Networks (2016, background): “, 2015b) and Grid LSTM (Kalchbrenner et al., 2016), there is no setting of the gates such that MU(h;W) simply returns the input h; the input is always processed with a non-linearity.”
Conditional Image Generation with PixelCNN Decoders (2016, background): “Two variants were proposed in the original paper: PixelRNN, where the pixel distributions are modeled with two-dimensional LSTM [7, 26], and PixelCNN, where they are modelled with convolutional networks.”
Recurrent Highway Networks (2016, methodology): “These problems have been recently addressed by architectures utilizing LSTM-based transformations for stacking (Zhang et al., 2016; Kalchbrenner et al., 2015).”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “imised cell that was the top performer among thousands of candidates. 5 Under review as a conference paper at ICLR 2018 Model Size Depth Valid Test Stacked LSTM, Graves (2013) 21M 7 - 1.67 Grid LSTM, Kalchbrenner et al. (2015) 17M 6 - 1.47 MI-LSTM, Wu et al. (2016) 17M 1 - 1.44 LN HM-LSTM, Chung et ”
Hierarchical Multiscale Recurrent Neural Networks (2016, neutral): “…1.27 LayerNorm HyperNetworks (Ha et al., 2016) (Graves, 2013) 1.67 MRNN (Sutskever et al., 2011) 1.60 GF-LSTM (Chung et al., 2015) 1.58 Grid-LSTM (Kalchbrenner et al., 2015) 1.47 MI-LSTM (Wu et al., 2016) 1.44 Recurrent Memory Array Structures (Rocki, 2016a) 1.40 SF-LSTM (Rocki, 2016b) ‡ 1.37…”
Neural Random Access Machines (2015, methodology): “Other models with this property include variants of Stack-Augmented recurrent neural networks (Joulin & Mikolov, 2015; Grefenstette et al., 2015), and the Grid-LSTM (Kalchbrenner et al., 2015)—of which the Grid-LSTM has achieved the greatest success on both synthetic and real tasks.”
Evolving Deep Neural Networks (2017, methodology): “Recent research on LSTMs has focused in two directions: Finding variations of individual LSTM memory unit architecture [4, 7, 25, 28], and discovering new ways of stitching LSTM layers into a network [8, 26, 53]. Both approaches have improved performance over vanilla LSTMs, with best recent results ”
Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “Recently there has been a resurgence of new structural designs for recurrent neural networks (RNNs) [1, 2, 3].”
Very deep convolutional networks for end-to-end speech recognition (2016, background): “Several architectures have been proposed recently to enable training of very deep networks [27, 23, 28, 29].”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, neutral): “…all experiments involved predicting discrete symbols (e.g. phonemes, characters, or words); as a result, the output of the decoder in each of our models was fed into an affine transformation followed by a softmax nonlinearity with a dimensionality corresponding to the number of possible…”