A comparison of statistical significance tests for information retrieval evaluation

By Mark D. Smucker, James Allan, Ben Carterette. 2007

787 citations recorded by the source corpus.

Information retrieval (IR) researchers commonly use three tests of statistical significance: the Student's paired t-test, the Wilcoxon signed rank test, and the sign test. Other researchers have previously proposed using both the bootstrap and Fisher's randomization (permutation) test as non-parametric significance tests for IR but these tests have seen little use. For each of these five tests, we took the ad-hoc retrieval runs submitted to TRECs 3 and 5-8, and for each pair of runs, we measured the statistical significance of the difference in their mean average precision. We discovered that there is little practical difference between the randomization, bootstrap, and t tests. Both the Wilcoxon and sign test have a poor ability to detect significance and have the potential to lead to false detections of significance. The Wilcoxon and sign tests are simplified variants of the randomization test and their use should be discontinued for measuring the significance of a difference between means.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

A Deep Relevance Matching Model for Ad-hoc Retrieval (2016, methodology): “Statistical differences between models are computed using the Fisher randomization test [24] (α = 0.”
Deformable part models are convolutional neural networks (2014, methodology): “016 under a paired-sample permutation test, a standard method from information retrieval [36].”
LSDA: Large Scale Detection through Adaptation (2014, neutral): “017 using a paired sample t-test [25]).”
Deep Learning Based Vulnerability Detection: Are We There Yet? (2020, result): “When comparing the results to baselines, we use statistical signiﬁ-cance test [51], [64], [65] and effect size test [66].”
Keyphrase Generation for Scientific Document Retrieval (2020, neutral): “05 (Smucker et al., 2007).”
Artificial error generation for translation-based grammatical error correction (2016, methodology): “The most common methods are randomisation tests and the bootstrap (Efron and Tibshirani, 1993), which have been found to largely agree with each other (Smucker et al., 2007).”
Test Collection Based Evaluation of Information Retrieval Systems (2010, methodology): “[235] who compared the Wilcoxon, sign, and t-tests with the bootstrap and the randomization or permutation test.”
Learning to Reweight Terms with Distributed Representations (2015, methodology): “05 rather than the Wilcoxon signed rank test that was used in prior research [1], which is prone to type I errors and considered not reliable for ad hoc IR [17, 18].”
Overview of the ShARe/CLEF eHealth Evaluation Lab 2014 (2014, methodology): “We relied on the Wilcoxon test [12] in Task 3 to better compare the measure values for the systems and benchmarks.”
Analysis of the Paragraph Vector Model for Information Retrieval (2016, methodology): “As suggested by Smucker et al.[21], statistical signiﬁcance is computed with Fisher randomization test with threshold 0.05.”
Semantic Matching by Non-Linear Word Transportation for Information Retrieval (2016, methodology): “Statistical differences between models are computed using the Fisher randomization test [29] (α = 0.”
Learning over Knowledge-Base Embeddings for Recommendation (2018, methodology): “Significant test is conducted based on the Fisher randomization test [41].”
Freebase QA: Information Extraction or Semantic Parsing? (2014, background): “4 The difference is not significant according to the paired permutation test (Smucker et al., 2007).”
Modeling and solving term mismatch for full-text retrieval (2012, result): “We draw different conclusions from that work (Smucker et al. 2007).”
Statistical reform in information retrieval? (2014, methodology): “distribution-free, computer-based alternatives to parametric significance testing, namely, the bootstrap test [16, 19] and the randomisation test [20, 22].”
Deep Relevance Ranking Using Enhanced Document-Query Interactions (2018, methodology): “To test for statistical significance between two systems, we employed twotailed stratified shuffling (Smucker et al., 2007; Dror et al., 2018) using the model with the highest”
Anytime Ranking for Impact-Ordered Indexes (2015, methodology): “Significance testing with respect to exhaustive evaluation (of all postings) was conducted using Fisher’s two-sided, paired randomization test [16]; following convention, ∗ denotes p < 0.”