LSTM: A Search Space Odyssey

By Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, J. Urgen Schmidhuber. 2015

6,287 citations recorded by the source corpus.

Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yoshua Bengio
  3. Jianfeng Gao
  4. Percy Liang
  5. Quoc V. Le
  6. Luke S. Zettlemoyer
  7. Kyunghyun Cho
  8. Graham Neubig

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Yi Yang
  8. Andrew Zisserman

In Model Architectures

  1. Yoshua Bengio
  2. Quoc V. Le
  3. Kyunghyun Cho
  4. Yoav Goldberg
  5. Richard Socher
  6. Mohit Bansal
  7. Ruslan Salakhutdinov
  8. Graham Neubig

In Signal Processing

  1. Yoshua Bengio
  2. Jinyu Li
  3. Dong Yu
  4. Tara N. Sainath
  5. Hung-yi Lee
  6. Yu Zhang
  7. James R. Glass
  8. Mirco Ravanelli

Citations in the corpus