The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing

By Rotem Dror, Gili Baumer, Segev Shlomov, Roi Reichart. 2018

487 citations recorded by the source corpus.

Statistical significance testing is a standard statistical tool designed to ensure that experimental results are not coincidental. In this opinion/ theoretical paper we discuss the role of statistical significance testing in Natural Language Processing (NLP) research. We establish the fundamental concepts of significance testing and discuss the specific aspects of NLP tasks, experimental setups and evaluation measures that affect the choice of significance tests in NLP research. Based on this discussion we propose a simple practical protocol for statistical significance test selection in NLP setups and accompany this protocol with a brief survey of the most relevant tests. We then survey recent empirical papers published in ACL and TACL during 2017 and show that while our community assigns great value to experimental results, statistical significance testing is often ignored or misused. We conclude with a brief discussion of open issues that should be properly addressed so that this important tool can be applied. in NLP research in a statistically sound manner.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Do Not Have Enough Data? Deep Learning to the Rescue! (2019, positive): “We statistically validated our results with the McNemar test (McNemar 1947; Dror et al. 2018).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “Regarding this, we encourage readers to conduct signiﬁcance testing (Dror et al., 2018) and report more substantial model improvements.”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, background): “The results are averaged after three runs to show the significance (Dror et al., 2018), each run takes around 5 hours.”
FLEX: Unifying Evaluation for Few-Shot NLP (2021, methodology): “Moreover, we newly advocate for controlling for the same sampled few-shot episodes across all methods and reporting individual episode results, so that researchers can run higher-powered paired statistical tests when comparing results [22], crucial when the benchmark has been optimized for low evalu”
With Little Power Comes Great Responsibility (2020, methodology): “Significance testing for BLEU: To test for a significant difference between two MT models we use the randomization test, as recommended in Dror et al. (2018): given the paired output translations from both models, swap the outputs for a random subset of test examples and compute the resulting…”
Large-Scale Multi-Label Text Classification on EU Legislation (2019, methodology): “For statistical significance tests, we take the run of each method with the best performance on development data, and perform two-tailed approximate randomization tests (Dror et al., 2018) on test data.8 Unless otherwise stated, we used 200-”
Latent Relation Language Models (2019, methodology): “Asterisk symbols represent statistical significance according to Wilcoxon signed-rank test (Dror et al., 2018) against the better model among NKLM and Vanilla LM, with p < 0.”
Can Large Language Models Transform Computational Social Science? (2023, background): “test (Dror et al., 2018)) are marked blue ; and † denote cases where zero-shot LLMs match or beat finetuned baselines.”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, methodology): “This signiﬁcance test ﬁts our task as F-score is generally assumed to be not normally distributed (Dror et al., 2018), and non-parametric signiﬁcance tests should be used in such a case.”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “Assuming the bootstrap (Efron and Tibshirani, 1993) which is a common procedure in NLP (Dror et al., 2018), we can estimate the expectation quantities in the decomposition.”
Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021, methodology): “We perform two statistical tests for verifying our results: i) The micro-sign test (s-test) (Yang and Liu, 1999), and ii) McNermar’s test (Dror et al., 2018).”
Fact or Fiction: Verifying Scientific Claims (2020, methodology): “We evaluate the statistical robustness of our results by generating 10,000 bootstrap-resampled versions of the test set (Dror et al., 2018) and computing the standard deviation of all performance metrics.”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, result): “In line with our reliability goal, just as statistical tests should appear in publications (Dror et al., 2018; Dodge et al., 2019), they should be “freebies” for leaderboard participants (Ethayarajh and Jurafsky, 2020).”
Equity Beyond Bias in Language Technologies for Education (2019, background): “Other work focuses on replication, allowing consistent tying of modeling choices to changes in behavior (Dror et al., 2017, 2018).”