Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

By Nils Reimers, Iryna Gurevych. 2017

312 citations recorded by the source corpus.

Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a "black art that requires expert experiences" (Snoek et al., 2012). In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Optimization & Training

Citations in the corpus

SciBERT: A Pretrained Language Model for Scientific Text (2019, background): “We did not ﬁnd changing the depth or size of the BiLSTMs to signiﬁcantly impact re-sults (Reimers and Gurevych, 2017).”
Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, background): “In addition, the hyperparameters of LSTM are just like RNN: the number of hidden layers, the number of units in each layer, network weight initialization, activation functions, learning rate, momentum values, the number of epochs, batch size (minibatch size), decay rate, optimization algorithms, seq”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, neutral): “Detailed information can be found in (Reimers and Gurevych, 2017).11”
Data Augmentation techniques in time series domain: a survey and taxonomy (2022, unknown): “Memory[85], Bidirectional Long Short-Term Memory[89] and Long Short-Term Memory Fully Convolutional Network[54].”
AliCoCo: Alibaba E-commerce Cognitive Concept Net (2020, background): “Figure 4 shows the principle architecture of a BiLSTMCRF model, which is the state-of-the-art model for various sequence labeling tasks [14, 23].”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, methodology): “This architecture (Figure 3) is similar to the one proposed by Reimers and Gurevych (2017).”
Improving time series forecasting using LSTM and attention models (2021, background): “As well, selecting a very high number of neurons may lead to overfitting (Reimers and Gurevych 2017).”
Transfer learning for biomedical named entity recognition with neural networks (2018, unknown): “However, there are opportunities to further optimize this approach, for instance, by determining the optimal size of the source dataset, developing robust methods of filtering noise from the source dataset, and extensive hyperparameter tuning (Young et al., 2015; Reimers and Gurevych, 2017).”
An empirical survey of data augmentation for time series classification with neural networks (2020, methodology): “The hyperparameters selected for the experiments were determined by the LSTM hyperparameter survey in [112].”
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, result): “BiLSTM-Softmax models perform inferior to BiLSTM-CRF models in most cases in the multilingual setting: The results show that the BiLSTM-CRF approach is stronger than the BiLSTM-Softmax approach on three of the four tasks, which are consistent with previous work on sequence labeling (Ma and Hovy, 201”