Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms

By Thomas G. Dietterich. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “In order to assess whether the difference between models at any given threshold reaches statistical significance, we use McNemar’s significance test, a paired, non-parametric test that has been advocated for the evaluation of binary classifiers (Dietterich, 1998).”
Improving Vector Space Word Representations Using Multilingual Correlation (2014, methodology): “01) according to the McNemar’s test (Dietterich, 1998).”
Distributional Memory: A General Framework for Corpus-Based Semantics (2010, methodology): “This approach to significance testing is problematic in many respects, the most important being that we ignore dependencies in correct and wrong counts due to the fact that the algorithms are evaluated on the same test set (Dietterich 1998).”
With Little Power Comes Great Responsibility (2020, methodology): “The standard statistical test for comparing classifiers on paired data is McNemar’s test (Dietterich, 1998; Dror et al., 2018), which uses the numbers of items where the models disagree (i.”
Multi-task learning for cross-platform siRNA efficacy prediction: an in-silico study (2010, methodology): “To verify the statistical significance of our model over the baseline algorithms, the paired t-test on the experimental results is also conducted [37].”
Robust Classification for Imprecise Environments (2000, methodology): “Applied statistics provides methods such as cross-validation and the bootstrap for estimating model error rates and recent studies have compared the effectiveness of different methods (Dietterich, 1998; Kohavi, 1995; Salzberg, 1997).”
KEEL: a software tool to assess evolutionary algorithms for data mining problems (2008, methodology): “To design an experiment, we have flrst to indicate the type of validation (k-fold cross validation [21] or 5x2 cross validation [ 12 ]) and the type of learning (regression, classiflcation or unsupervised) to be used.”
Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks (2019, background): “For example, RF models report a year-agnostic mortality AUROC of 0.82± 0.02 (5 x 2 fold CV splits (Dietterich, 1998)), as compared to their true year-averaged AUROC under the raw representation of 0.76± 0.13.”
Random forests as a tool for ecohydrological distribution modelling (2007, methodology): “Based on the conclusions of Dietterich (1998), the McNemar test (Everitt, 1992) was selected to compare the performances of the multiple logistic regression model and the random forest model.”
Profit Driven Decision Trees for Churn Prediction (2017, methodology): “6: five replications of 2-fold cross-validation on the training set (Dietterich, 1998; Demšar, 2006).”
Drug drug interaction extraction from biomedical literature using syntax convolutional neural network (2016, methodology): “The improvements have been proven to be significant using MCNemar’s test (Dietterich, 1998) at the 0.05 level.”
Benchmarking Least Squares Support Vector Machine Classifiers (2004, methodology): “Since the observations on the randomizations are not independent ( Dietterich, 1998 ), we remark that this standard t-test is used only as a (common) heuristic to indicate statistical difference between average accuracies on the ten randomizations.”
High-throughput prediction of protein antigenicity using protein microarray data (2010, methodology): “2.3.2 Evaluation and comparison with previous methods Repeated crossvalidations are usually recommended to obtain reliable accuracy estimates (Dietterich, 1998; Kohavi, 1995).”