Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis

By Alessio Benavoli, Giorgio Corani, Jaka Demsar, Marco Zaffalon. Journal of machine learning research, 2016

496 citations recorded by the source corpus.

The machine learning community adopted the use of null hypothesis significance testing (NHST) in order to ensure the statistical validity of results. Many scientific fields however realized the shortcomings of frequentist reasoning and in the most radical cases even banned its use in publications. We should do the same: just as we have embraced the Bayesian paradigm in the development of new machine learning methods, so we should also use it in the analysis of our own results. We argue for abandonment of NHST by exposing its fallacies and, more importantly, offer better - more sound and useful - alternatives for it.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/8ce2c4a374e8b37e3eef080c956f22cfc6ea25d6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SemEval-2018 Task 12: The Argument Reasoning Comprehension Task (2018, methodology): “The machine learning community has already abandoned the controversial p-value and replaced it with Bayesian methods that are easily interpretable and account well for uncertainty (Benavoli et al., 2017).”
Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System (2019, methodology): “To further evaluate the statistical significance of the similarity between the results of CF and FCF, we performed a Bayesian correlated t-test [29, 30].”
Recent Trends in the Use of Statistical Tests for Comparing Swarm and Evolutionary Computing Algorithms: Practical Guidelines and a Critical Review (2020, methodology): “The most repeated criticisms about the NHST are: • They do not provide the answer that we expected [76, 8].”
A Tutorial On the design, experimentation and application of metaheuristic algorithms to real-World optimization problems (2021, background): “On the contrary, considerable diﬀerences could not yield small p-values if there are not enough data (as datasets) for testing the method [108].”
Evaluating time series forecasting models: an empirical study on performance estimation methods (2019, methodology): “of practical equivalence [2] (ROPE) to be the interval [-2.”