Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms

By Remco R. Bouckaert, Eibe Frank. 2004

443 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Joelle Pineau
3. Doina Precup
4. Pieter Abbeel
5. Peter Henderson
6. Yoshua Bengio
7. Shixiang Shane Gu
8. David Meger

In Statistics and Probability

1. Peter A. Flach
2. Bernhard Pfahringer
3. Michael J. Cree
4. Henry Gouk
5. Philip Bachman
6. Doina Precup
7. Joelle Pineau
8. Eibe Frank

In Evaluation & Benchmarks

1. Peter A. Flach
2. Bernhard Pfahringer
3. Eibe Frank
4. Andreas Zell
5. Nikolas Fechner
6. Georg Hinselmann
7. Lars Rosenbaum
8. Andreas Jahn

Citations in the corpus

• Deep Reinforcement Learning that Matters (2017, methodology): “Similar lines of investigation to (Nadeau and Bengio 2000; Bouckaert and Frank 2004) would be helpful to determine the best methods for evaluating performance gain significance.”
• Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “We use the hypothesis testing procedure outlined by Bouckaert and Frank (2004) in order to overcome the problem of overlapping testing and training sets introduced by the cross validation procedure.”
• The Scientific Method in the Science of Machine Learning (2019, unknown): “Neal (1998); Dietterich (1998); Nadeau & Bengio (2000); Bouckaert & Frank (2004) provided ML researchers with methods of statistical comparison across experimental results on supervised tasks.”
• An Introduction to Deep Reinforcement Learning (2018, unknown): “a reasonable benchmark strategy, techniques derived from significance testing (Demšar, 2006; Bouckaert and Frank, 2004; Bouckaert, 2003; Dietterich, 1998) have the advantage of providing statistically grounded arguments in favor of a given hypothesis.”
• jCompoundMapper: An open source Java library and command-line tool for chemical fingerprints (2011, methodology): “Based on these statistics, the corrected resampled t -test of Bouckaert and Frank [32] can be applied.”
• Machine Learning - The Art and Science of Algorithms that Make Sense of Data (2012, methodology): “Experimental studies regarding cross-validation are carried out by Dietterich (1998) and Bouckaert and Frank (2004): the former recommends five times two-fold cross-validation and the latter ten times ten-fold.”