LSTM: A Search Space Odyssey
By Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, J. Urgen Schmidhuber. 2015
6,287 citations recorded by the source corpus.
Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
- Yoav Goldberg
- Richard Socher
- Mohit Bansal
- Ruslan Salakhutdinov
- Graham Neubig
In Signal Processing
Citations in the corpus
- Massive Exploration of Neural Machine Translation Architectures (2017, background): “While there exist studies (Greff et al., 2016) that explore cell variants on small sequence tasks of a few thousand examples, we are not aware of such studies in large-scale NMT settings.”
- Semi-supervised Sequence Learning (2015, background): “Our LSTM implementation is standard and has input gates, forget gates, and output gates [6, 7, 8].”
- Character-level Convolutional Networks for Text Classification (2015, methodology): “The variant of LSTM we used is the common “vanilla” architecture [8] [9].”
- Learning Recurrent Span Representations for Extractive Question Answering (2016, methodology): “We couple the input and forget gates in our LSTMs, as described in Greff et al. (2016), and we use a single dropout mask to apply dropout across all LSTM time-steps as proposed by Gal & Ghahramani (2016).”
- Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, neutral): “LSTM (standard two-layer) (Greff et al., 2016) and CNN-CRF (standard two-layer) (Chu et al., 2016) yield comparable results (4.4-4.6%).”
- Light Gated Recurrent Units for Speech Recognition (2018, background): “purpose RNNs such as Long Short Term Memories (LSTMs) [28] have been the subject of several studies and modifications over the past years [7], [29], [30].”
- On Network Design Spaces for Visual Recognition (2019, background): “In the absence of rigorous theoretical understanding of deep networks, it is imperative to perform large-scale empirical studies of deep networks to aid development [6, 3, 27].”
- Neural Responding Machine for Short-Text Conversation (2015, methodology): “We use GRU in this work, since it performs comparably to LSTM on squence modeling (Chung et al., 2014; Greff et al., 2015), but has less parameters and easier to train.”
- Towards Universal Paraphrastic Sentence Embeddings (2015, background): “…variants, culminating in LSTMs because they have been found to be effective for many types of sequential data (Graves et al., 2008; 2013; Greff et al., 2015), including text (Sutskever et al., 2014; Vinyals et al., 2014; Xu et al., 2015a; Hermann et al., 2015; Ling et al., 2015; Wen et al., 2015).”
- Recurrent Highway Networks (2016, background): “It reduces model size for a fixed number of units and prevents an unbounded blow-up of state values leading to more stable training, but imposes a modeling bias which may be suboptimal for certain tasks (Greff et al., 2015; Jozefowicz et al., 2015).”
- Visualizing and Understanding Recurrent Networks (2015, background): “A few recent ablation studies analyzed the effects on performance as they removed or modified various gates and connections (12; 5).”
- Generation and Comprehension of Unambiguous Object Descriptions (2015, methodology): “We adopt the most commonly used vanilla LSTM structure [21] and feed the visual representation as input to the LSTM at each time step.”
- Understanding Neural Networks through Representation Erasure (2016, background): “, 2015), and empirical studies of LSTM components (Greff et al., 2015; Chung et al., 2014).”
- Deep Biaffine Attention for Neural Dependency Parsing (2016, methodology): “We also implemented the coupled input-forget gate LSTM cells (Cif-LSTM) suggested by Greff et al. (2015), 6 finding that while the resulting model still slightly underperforms the more popular LSTM cells, the difference between the two is much smaller.”
- Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, background): “Equations 11-15 show the form of the forward pass of the LSTM unit [58] (xt: input vector to the LSTM unit, ft: forget gate’s activation vector, it: input gate’s activation vector, ot: output gate’s activation vector, ht: output vector of the LSTM unit, ct: cell state vector, σg: sigmoid function, σ”
- Generative Models for Effective ML on Private, Decentralized Datasets (2019, methodology): “DP Char-LM The model architecture used to train the DP char-LM is based on “FLML ” in Chen et al. (2019), which is also a CIFG-LSTM with projection using 3 layers, 256 RNN hidden units and 128 projected dimensions.”