Cost-conscious comparison of supervised learning algorithms over multiple data sets

By Aydın Ulaş, Olcay Taner Yildiz, Ethem Alpaydin. 2012

49 citations recorded by the source corpus.

In the literature, there exist statistical tests to compare supervised learning algorithms on multiple data sets in terms of accuracy but they do not always generate an ordering. We propose Multi^2Test, a generalization of our previous work, for ordering multiple learning algorithms on multiple data sets from ''best'' to ''worst'' where our goodness measure is composed of a prior cost term additional to generalization error. Our simulations show that Multi^2Test generates orderings using pairwise tests on error and different types of cost using time and space complexity of the learning algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Francisco Herrera
2. Salvador Garcia
3. Swagatam Das
4. María del Mar Rueda
5. Jacinto Carrasco
6. Vanya Van Belle
7. Lieven Billiet
8. S. Van Huffel

In Computer Vision and Pattern Recognition

1. Salvador Garcia
2. Francisco Herrera
3. S. Van Huffel
4. Vanya Van Belle
5. Lieven Billiet

Citations in the corpus

• Interval Coded Scoring: a toolbox for interpretable scoring systems (2018, methodology): “The Bergmann-Hommel procedure was applied to the overall significance level of 0.05 as posthoc correction for the influence of multiple comparison (Ulas¸ et al., 2012).”
• Recent Trends in the Use of Statistical Tests for Comparing Swarm and Evolutionary Computing Algorithms: Practical Guidelines and a Critical Review (2020, methodology): “[38] Non-parametric - Classification Multi(2)Test algorithm, which orders the algorithms using non-parametric pairwise tests.”