A Plea for Neutral Comparison Studies in Computational Sciences

By Anne-Laure Boulesteix, Manuel J. A. Eugster. 2012

134 citations recorded by the source corpus.

In computational science literature including, e.g., bioinformatics, computational statistics or machine learning, most published articles are devoted to the development of “new methods”, while comparison studies are generally appreciated by readers but surprisingly given poor consideration by many journals. This paper stresses the importance of neutral comparison studies for the objective evaluation of existing methods and the establishment of standards by drawing parallels with clinical research. The goal of the paper is twofold. Firstly, we present a survey of recent computational papers on supervised classification published in seven high-ranking computational science journals. The aim is to provide an up-to-date picture of current scientific practice with respect to the comparison of methods in both articles presenting new methods and articles focusing on the comparison study itself. Secondly, based on the results of our survey we critically discuss the necessity, impact and limitations of neutral comparison studies in computational sciences. We define three reasonable criteria a comparison study has to fulfill in order to be considered as neutral, and explicate general considerations on the individual components of a “tidy neutral comparison study”. R codes for completely replicating our statistical analyses and figures are available from the companion website http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/plea2013.

Subfields: Molecular Biology, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, background): “However, while the question of reproducibility and good experimental practice has been examined in related ﬁelds (Wagstaff 2012; Boulesteix, Lauer, and Eugster 2013; Stodden, Leisch, and Peng 2014; Bouckaert and Frank 2004; Bouckaert 2004; Vaughan and Wawerla 2012), to the best of our knowledge…”
NetCoMi: network construction and comparison for microbiome data in R (2020, unknown): “Finally, we posit that simulation studies accompanying articles that introduce new methods might also be inherently biased [118, 119].”
Random forest versus logistic regression: a large-scale benchmark experiment (2018, background): “Further discussions of the concept of authors’ neutrality can be found elsewhere [5, 11].”
Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications (2017, unknown): “In light of the increasing interest in DL, however, we cannot exclude a publication bias which favoured studies showing the superiority of this new analytical approach relative to conventional shallow ML methods (Boulesteix et al., 2013).”
A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. (2019, methodology): “Studies were excluded if a new modeling approach was introduced (hence a methodological focus) [20,21], models were developed for nonhumans, the models made predictions for individual images or signals rather than participants, models were developed based on high-dimensional data modalities, the…”
Ten Simple Rules for Reducing Overoptimistic Reporting in Methodological Computational Research (2015, methodology): “To establish that a new method works well in practice, it is important to evaluate its performance using several datasets, just as it is important to evaluate the efficiency of a new drug based on several patients before recommending it for use on other patients [5,8].”