Should We Really Use Post-Hoc Tests Based on Mean-Ranks?

By Alessio Benavoli, Giorgio Corani, Francesca Mangili. Journal of machine learning research, 2015

470 citations recorded by the source corpus.

The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. The multiple comparisons are usually performed using the mean-ranks test. The aim of this technical note is to discuss the inconsistencies of the mean-ranks post-hoc test with the goal of discouraging its use in machine learning as well as in medicine, psychology, etc. We show that the outcome of the mean-ranks test depends on the pool of algorithms originally included in the experiment. In other words, the outcome of the comparison between algorithms A and B depends also on the performance of the other algorithms included in the original experiment. This can lead to paradoxical situations. For instance the difference between A and B could be declared significant if the pool comprises algorithms C, D, E and not significant if the pool comprises algorithms F, G, H. To overcome these issues, we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1435c09304f24a1310e6ede11bf5dfb4dbe78c3d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

HIVE-COTE 2.0: a new meta ensemble for time series classification (2021, methodology): “For multiple classifiers over multiple datasets we use an adaptation of the critical difference diagram (Demšar, 2006), replacing the post-hoc Nemenyi test with a comparison of all classifiers using pairwise Wilcoxon signed-rank tests, and cliques formed using the Holm correction recommended by (Gar”
InceptionTime: Finding AlexNet for time series classification (2019, methodology): “We then performed the pairwise post-hoc analysis recommended by Benavoli et al. (2016) where we replaced the average rank comparison by a Wilcoxon signed-rank test with Holm’s alpha (5”
Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis (2016, methodology): “In Bayesian estimation, this can be obtained by deﬁning a hierarchical model (Corani et al., 2016).”
The UCR time series archive (2018, methodology): “We use the Friedmann test [39], [40] to determine if there were any statistically significant differences in the rankings of the classifiers.”
Deep learning for time series classification: a review (2018, methodology): “Then we performed the pairwise posthoc analysis recommended by Benavoli et al. (2016) where the average rank comparison is replaced by a Wilcoxon signed-rank test (Wilcoxon, 1945) with Holm’s alpha (5”
MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification (2020, unknown): “line [9, 12, 4].”
The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances (2020, methodology): “Based on the literature (Benavoli et al. 2016), we abandon the post hoc test used in Demšar (2006) and instead form cliques with pairwise tests, making the Holm correction for multiple testing.”
ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels (2019, methodology): “…which the diﬀerence in pairwise classiﬁcation accuracy is not statistically signiﬁcant, as determined by a Wilcoxon signed-rank test with Holm correction (as a post hoc test to the Friedman test), are connected with a black line (see Demˇsar 2006; Garc´ıa and Herrera 2008; Benavoli et al. 2016).”