Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

By Nils Reimers, Iryna Gurevych. 2017

470 citations recorded by the source corpus.

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^{-4}) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F₁-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that produce both superior performance as well as are more stable with respect to the remaining hyperparameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Most sequence taggers and dependency parsers in prior work use small LSTMs (hidden state sizes of around 300) because larger models yield little to no gains in performance (Reimers and Gurevych, 2017).”
SciBERT: A Pretrained Language Model for Scientific Text (2019, methodology): “We did not find changing the depth or size of the BiLSTMs to significantly impact results (Reimers and Gurevych, 2017).”
Show Your Work: Improved Reporting of Experimental Results (2019, background): “Moreover, recent investigations into “state-ofthe-art” claims have found competing methods to only be comparable, without clear superiority, even against baselines (Reimers and Gurevych, 2017; Lucic et al., 2018; Li and Talwalkar, 2019); this has exposed the need for reporting more than a single poi”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, background): “As shown in our publication (Reimers and Gurevych, 2017), presenting a single configuration together with the achieved performance is not meaningful.”
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2019, background): “Several past works have noted that the same architecture can perform very differently across random restarts (Reimers and Gurevych, 2017, 2018).”
Quantity doesn’t buy quality syntax with neural language models (2019, neutral): “The limited effect of network size is consistent with previous findings on sequence labeling tasks (Reimers and Gurevych, 2017; Greff et al., 2017).”
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning (2020, methodology): “…appendix can be aggregated across multiple experiments to show averages along with standard error as recommended in prior work (Henderson et al., 2018; Colas et al., 2018; Reimers and Gurevych, 2017). from experiment_impact_tracker. compute_tracker import ImpactTracker tracker = ImpactTracker(”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, background): “In (Reimers and Gurevych, 2017, 2018), we showed that the random seed can have a large impact on the performances of trained models, especially for small datasets.”
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (2018, unknown): “Reporting single evaluation scores might result in biased results (Reimers and Gurevych, 2017).”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, methodology): “Being aware of the infamous instability of neural approaches (Reimers and Gurevych, 2017), we also employ a ridge regression model, an L2 regularized version of linear regression, as a more robust, yet also powerful baseline (Li et al., 2017).”
Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019, background): “It is known that multiple runs of probabilistic deep learning models can have variance in overall scores (Reimers and Gurevych, 2017) 13 .”
Multilingual Neural Machine Translation with Task-Specific Attention (2018, background): “Each set of parameters provides different levels of generalization (Reimers and Gurevych, 2017), which is evidenced in the synergistic task of training multilingual translation models.”
Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction (2018, background): “BiLSTM-CNN-CRF (Reimers and Gurevych, 2017) is the state-of-the-art from the Named Entity Recogntion (NER) community.”
Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding (2018, background): “Reimers and Gurevych (2017) pointed out that neural network training is nondeterministic and depends on the seed for the random number generator.”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, methodology): “Each set of parameters provides a different level of generalization (Reimers and Gurevych, 2017).”
Pooled Contextualized Embeddings for Named Entity Recognition (2019, methodology): “For our experiments, we follow the training and evaluation procedure outlined in Akbik et al. (2018) and follow most hyperparameter suggestions as given by the in-depth study presented in Reimers and Gurevych (2017).”
We need to talk about random seeds (2022, unknown): “It is a good idea to compare not just the point estimate of a single model’s performance, but distributions of model performance, as comparing performance distributions results in more reliable conclusions (Reimers and Gurevych, 2017; Dodge et al., 2019; Radosavovic et al., 2020).”
Graph-to-Sequence Learning using Gated Graph Neural Networks (2018, background): “Recent work has shown that evaluation in neural models can lead to wrong conclusions by just changing the random seed (Reimers and Gurevych, 2017).”