Evaluating Learning Algorithms: A Classification Perspective

By Nathalie Japkowicz, Mohak Shah. 2011

1,037 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RGB-D-based action recognition datasets: A survey (2016, neutral): “Perhaps, in line with protocols of well designed statistical experiments, the results reported for action recognition algorithms should also include statistical significance tests [112].”
Heterogeneous Feature Models and Feature Selection Applied to Bearing Fault Diagnosis (2015, methodology): “rigor and allow comparison, it is necessary to use statistically sound and robust methods [33].”
Algorithmic Accountability and Public Reason (2017, background): “These include questions which are internal to the practice of algorithm design and machine learning, such as whether a model is generalizable, over-fitted or overtrained, and other questions relating to the performance of machine learning algorithms (Japkowicz & Shah, 2011).”
Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data (2017, background): “Diﬀerent evaluation mechanisms for models emphasise diﬀerent aspects of performance (Japkowicz and Shah, 2011).”
Comprehensible classification models: a position paper (2014, background): “The vast majority of works on classification model evaluation use predictive accuracy as the only evaluation criterion [34], [36],”
A network intrusion detection system based on a Hidden Naïve Bayes multiclass classifier (2012, background): “These two measures effectively summarize the overall performance by considering all of the classes and generalizing the classiﬁer performance in terms of the convergence behaviors (Japkowicz & Shah, 2011).”
α-Rank: Multi-Agent Evaluation by Evolution (2019, background): “Researchers have also carried out studies of evaluation metrics in the fields of computer science, machine learning, and artificial intelligence [90–92].”
Analysing the Overfit of the Auto-sklearn Automated Machine Learning Tool (2019, methodology): “We use the popular Area Under the Receiver Operating Characteristic curve (AUROC) to measure the predictive accuracy of auto-sklearn [5].”
Learning under Concept Drift: A Review (2019, methodology): “The three most frequently used statistical tests for comparing two learning algorithms [96], [97] are: 1) McNemar test [98]: denote the number of data instances misclassified by the first classifier and correctly classified by the second classifier by a, and denote b in the opposite way.”
Ordered Decompositional DAG kernels enhancements (2015, methodology): “A 10x10 CV test with confidence level 95% (and 10 degrees of freedom) has been executed between each pair of kernels on the two datasets [44].”
Artificial Intelligence and Patient-Centered Decision-Making (2020, background): “As of yet, there are no standardized performance metrics for machine learning systems, which makes it hard to compare these systems with each other and with human practitioners (Japkowicz and Shah 2011).”
Comparison and benchmark of name-to-gender inference services (2018, methodology): “Since our data set is relatively large and the550 data sets have been obtained by random sampling, one needs to show that the assumption of homogeneity551 of variance is satisfied prior to applying a t-test (see e.g. Japkowicz and Shah (2014)[p. 222ff]).”
On evaluating stream learning algorithms (2012, background): “The design of experimental studies is of paramount importance (Japkowicz and Shah 2011).”
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement (2017, methodology): “Some previous works discussing AI evaluation (Newell and Simon, 1976; Gaschnig et al, 1983; Rothenberg et al, 1987; Geissman and Schultz, 1988; Decker et al, 1989; Langley, 1987, 2011; Buchanan, 1988; Simon, 1995; Baldwin and Yadav, 1995; Falkenauer, 1998; Langford, 2005; Legg and Hutter, 2007a; Whi”