More accurate tests for the statistical significance of result differences

By Alexander S. Yeh. 2000

464 citations recorded by the source corpus.

Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly used tests often underestimate the significance and so are less likely to detect differences that exist between different techniques. This underestimation comes from an independence assumption that is often violated. We point out some useful tests that do not make this assumption, including computationally-intensive randomization tests.

Subfields: Artificial Intelligence, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “We test the significance of the results using randomisation tests (Yeh, 2000) with Bonferroni correction (Abdi, 2007).”
An MDP-Based Recommender System (2002, neutral): “, Yeh (2000)) showed that mean1/mean2 was greater than one with probability 0.”
A Structured Vector Space Model for Word Meaning in Context (2008, methodology): “Differences between the performance of models were tested for significance using a stratified shuffling-based randomization test (Yeh, 2000).”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, methodology): “We run an approximate randomization test by stratifying the output and permuting within each stratum (Yeh, 2000) and find that, for four attractors, the performance difference between the top-down RNNG and the other variants is statistically significant at p 0.05.”
Shallow Parsing with Conditional Random Fields (2003, background): “Yeh (2000) examined randomized tests for estimating the significance of F scores, and in particular the bootstrap over the test set (Efron and Tibshirani, 1993; Sang, 2002).”
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018, methodology): “For other evaluation measures, such as F-score, BLEU, METEOR and ROUGE that do not compute means, the common practice is to assume that they are not normally distributed (Yeh, 2000; Berg-Kirkpatrick et al., 2012).”
Abstractive Summarization of Spoken and Written Conversations Based on Phrasal Queries (2014, methodology): “The statistical significance tests was calculated by approximate randomization, as described in (Yeh, 2000).”
Lexical Normalisation of Short Text Messages: Makn Sens a #twitter (2011, background): “05 : Yeh (2000)), indicating the complementarity of the two methods, especially on Twitter data.”
Efficient Higher-Order CRFs for Morphological Tagging (2013, methodology): “Throughout the paper we establish significance by running approximate randomization tests on sentences (Yeh, 2000).”
Metrics for Evaluation of Word-level Machine Translation Quality Estimation (2016, methodology): “We rank the systems according to all the metrics and compute the level of signiﬁcance for every 2 Systems that took part in the shared task are listed and described in (Bojar et al., 2015). pair of systems with randomisation tests (Yeh, 2000) with Bonferroni correction (Abdi, 2007).”
Portable Automatic Text Classification for Adverse Drug Reaction Detection via Multi-corpus Training (2014, methodology): “Adding the ADE training instances to either of the other two data sets also 23Statistical significance of the F-score measured using the technique proposed by [54].”
A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art (2013, background): “As reported by Yeh (2000), other tests such as randomized testing, may also be appropriate.”
Sentiment Analysis of Short Informal Texts (2014, unknown): “05 level (Yeh, 2000).”
SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge (2013, methodology): “Statistical significance tests were conducted using approximate randomization test (Yeh, 2000) with 10,000 iterations; p ≤ 0.”
Have You Lost the Thread? Discovering Ongoing Conversations in Scattered Dialog Blocks (2017, methodology): “Finally, statistical significance was computed with Pado’s (2006) implementation of approximate randomization (Yeh 2000).”
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge (2018, methodology): “We performed pairwise significance tests using an approximate randomization test (Yeh, 2000) over texts.”