Benchmarking of LSTM Networks

By Thomas Breuel. 2015

54 citations recorded by the source corpus.

LSTM (Long Short-Term Memory) recurrent neural networks have been highly successful in a number of application areas. This technical report describes the use of the MNIST and UW3 databases for benchmarking LSTM networks and explores the effect of different architectural and hyperparameter choices on performance. Significant findings include: (1) LSTM performance depends smoothly on learning rates, (2) batching and momentum has no significant effect on performance, (3) softmax training outperforms least square training, (4) peephole units are not useful, (5) the standard non-linearities (tanh and sigmoid) perform best, (6) bidirectional training combined with CTC performs better than other methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Model Architectures, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jaehoon Lee
2. Jascha Narain Sohl-Dickstein
3. C. Pal
4. Yoshua Bengio
5. Greg Yang
6. Percy Liang
7. Quoc V. Le
8. Ruslan Salakhutdinov

In Computer Networks and Communications

1. Joseph M. Antognini
2. Jascha Narain Sohl-Dickstein
3. C. Pal
4. Christopher J. Shallue
5. Félix G. Harvey
6. Roy Frostig
7. George E. Dahl
8. Jaehoon Lee

In Model Architectures

1. Jascha Narain Sohl-Dickstein
2. Jaehoon Lee
3. Joseph M. Antognini
4. C. Pal
5. Christopher J. Shallue
6. Félix G. Harvey
7. Roy Frostig
8. George E. Dahl

In Evaluation & Benchmarks

1. Joseph M. Antognini
2. Jascha Narain Sohl-Dickstein
3. C. Pal
4. Christopher J. Shallue
5. Félix G. Harvey
6. Roy Frostig
7. George E. Dahl
8. Jaehoon Lee

Citations in the corpus

• Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “They found that batch size had no effect on solution quality for LSTM networks (Breuel, 2015a), but found that larger batch sizes achieved worse solutions for fully connected and convolutional networks, and that the scale of the effect depended on the activation function in the hidden and output…”
• A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition (2016, unknown): “Index Terms— acoustic modeling, LSTM, RNN”
• Semi-supervised Learning with Encoder-Decoder Recurrent Neural Networks: Experiments with Motion Capture Sequences (2015, methodology): “Such changes in the architecture are analysed in section 4.3.”
• Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks (2018, unknown): “provement has shown in recent experiments [40].”