Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement
By George Forman, Martin Scholz. SKDD, 2010
424 citations recorded by the source corpus.
Subfields: Artificial Intelligence, Information Systems
View Paper: https://www.semanticscholar.org/paper/647412df4f7e8ac2deb3daa9d0d943c140c54094
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tuomas Virtanen
- Iryna Gurevych
- Christian Stab
- Ekaterina Shutova
- Helen Yannakoudakis
- Pushkar Mishra
- Annamaria Mesaros
- Toni Heittola
In Information Systems
- Iryna Gurevych
- Martin J. Shepperd
- Tracy Hall
- David Bowes
- Christian Stab
- Jesse Davis
- Annamaria Mesaros
- Bhiksha Raj
Citations in the corpus
- DCASE2017 Challenge Setup: Tasks, Datasets and Baseline System (2017, methodology): “This method of calculating performance gives equal weight to each individual sound instance in each segment, as opposed to being influenced by class balance and error types [29].”
- Neural Character-based Composition Models for Abuse Detection (2018, background): “A possible explanation of this lies in the authors’ decision to not use stratification, which for such a highly imbalanced dataset can lead to unexpected outcomes (Forman and Scholz, 2010).”
- A multi-room reverberant dataset for sound event localization and detection (2019, methodology): “The performance metrics are calculated by accumulating the required statistics from all the folds [7], and not as the average of the metrics of the individual folds.”
- Unachievable Region in Precision-Recall Space and Its Effect on Empirical Evaluation (2012, background): “Unfortunately, this is rarely a primary goal of machine learning and learned models tend to be poorly calibrated (Forman & Scholz, 2010).”
- Researcher Bias: The Use of Machine Learning in Software Defect Prediction (2014, background): “The Supplementary Material contains an exploration of some alternative approaches.”
- TUT database for acoustic scene classification and sound event detection (2016, methodology): “The results from all folds were combined to produce a single evaluation, for avoiding biases caused by data imbalance between folds [22].”
- Identifying Argumentative Discourse Structures in Persuasive Essays (2014, methodology): “We determine the evaluation scores by accumulating the confusion matrices of each fold into one confusion matrix, since it is the less biased method for evaluating cross-validation studies (Forman and Scholz, 2010).”
- Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection (2017, neutral): “This calculation method is referred to as micro-averaging, and is the recommended method for evaluation of classifier [42].”
- Combining embedded accelerometers with computer vision for recognizing food preparation activities (2013, methodology): “…and recall over all classes and cross-validation partitions are computed by first summing TP, FP and FN over all partitions for each class separately, then applying the formulas for precision and recall on the sums and finally estimating the arithmetic mean over all classes, as argued in [6].”