Statistical Comparisons of Classifiers over Multiple Data Sets

By Jaka Demsar. 2006

13,306 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Following prior work by Demsar (2006), this enables computing statistical significance of numbers.”
Multi-Source Domain Adaptation: A Causal View (2015, methodology): “We use both the Wilcoxon signed ranks test and Friedman test, recommended by (Dems̆ar 2006), for performance comparison.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “We measure statistical significance with unpaired Wilcoxon (Demšar, 2006) signed-rank tests of CLUE vs each baseline.”
Interval Coded Scoring: a toolbox for interpretable scoring systems (2018, methodology): “Furthermore, Figure 7 shows a graphical representation of the multiple comparison test (Demˇsar, 2006).”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “These results can also be visualised using a critical diﬀerence diagram (Demˇsar, 2006), as shown in Figure 4.”
Meta-Learning: A Survey (2018, neutral): “We can then build a ranking per task, for instance using success rates, AUC, or significant wins (Brazdil et al., 2003a; Demšar, 2006; Leite et al., 2012).”
Obtaining Well Calibrated Probabilities Using Bayesian Binning (2015, methodology): “Next, we briefly describe the test procedure; more detailed information can be found in (Demšar 2006).”
With Little Power Comes Great Responsibility (2020, methodology): “It is also worth exploring power with respect to claims of improvement on multiple tasks with a single model (Demšar, 2006), rather than each task individually.”
Sequential Model-Based Ensemble Optimization (2014, methodology): “Hence, we evaluate our methods on several data sets using metrics that do not assume commensurability across tasks [Demšar, 2006].”
Finding a short and accurate decision rule in disjunctive normal form by exhaustive search (2010, methodology): “To compare the algorithms over multiple datasets we used a Friedman test with a conservative Bonferroni-Dunn post-hoc test as recently suggested by Demšar (2006). The Friedman test showed that there is a statistically significant difference in accuracy among the algorithms (p = 0.”
The jMetal framework for multi-objective optimization: Design and architecture (2010, methodology): “TheWilcoxon Textcheckbox in the bottom part of the GUI, if selected, generates R scripts to apply the Wilcoxon rank sum test, a non-parametric statistical hypothesis test, wh ich allows to make pairwise comparisons between algorithms to know about the significance of the obtained data [46].”
A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring (2017, background): “The core of SVM is to map the original training set into a high-dimensional feature space, in which non-linear separated features are replaced by a linear discriminant function in high-dimension (Cortes & Vapnik, 1995).”
An extensive experimental survey of regression methods (2019, methodology): “Table 16 reports the results of several statistical tests [75] developed to 374 compare the globally best regressor (cubist) and the 20 bests regressors in 375 terms of R(2).”