Why rankings of biomedical image analysis competitions should be interpreted with care

By Lena Maier-Hein, M. Eisenmann, Annika Reinke, Sinan Onogur, M. Stankovic, Patrick Godau, Tal Arbel, H. Bogunovic, Andrew P. Bradley, A. Carass and 28 more. Nature Communications, 2018

393 citations recorded by the source corpus.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future. Biomedical image analysis challenges have increased in the last ten years, but common practices have not been established yet. Here the authors analyze 150 recent challenges and demonstrate that outcome varies based on the metrics used and that limited information reporting hampers reproducibility.

Subfields: Biomedical Engineering, Health Informatics, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/91d23b702b9a59bf75c5162c3250017b526c0e69

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge (2019, methodology): “The use of common benchmarks such as this have a long history in predictive learning (West et al., 1997; Maier-Hein et al., 2018), but they first started to attract broader interest within the medical imaging community in 2007 when “Biomedical Grand Challenges” were first officially affiliated with ”
Causality matters in medical imaging (2019, methodology): “Based on a comprehensive ontology of medical imaging meta-information 11 , we have compiled in Table 1 a list of attributes that are meaningful for characterizing the predictive causal direction and detecting signs of dataset mismatch.”
Metrics Reloaded: Recommendations for image analysis validation (2022, neutral): “[58] Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P Bradley, Aaron Carass, et al.”
On the Interpretability of Artificial Intelligence in Radiology: Challenges and Opportunities. (2020, methodology): “Beyond transparency of the AI system itself, enhanced transparency of the evaluation procedures of AI technologies in biomedical imaging has also been highlighted by proposing guidelines and best-practice recommendations (39).”
Benchmarking atlas-level data integration in single-cell genomics (2020, neutral): “follows best practices for robust ranking in machine learning tasks 32 and indeed produced consistent overall rankings when compared to alternatives 31 (overall rank correlation, Spearman’s R > 0.96 for all tasks).”
Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis (2020, background): “The same tendency is observed in many of the datasets included in a recent analysis of 150 databases from grand challenges on biomedical image analysis (26), which provides recommendations for database and challenge design, where there is no explicit mention of the importance of sex/gender demograph”
Automated Design of Deep Learning Methods for Biomedical Image Segmentation (2019, background): “tion U-Net Generalization 1 Introduction Semantic segmentation remains a popular research topic in the domain of medical image computing with 70% of the international competitions being devoted to it [8]. Important reasons for the enduring interest certainly are the diversity and individual peculiar”
Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “, multi-scale and deformations, [7], the necessity to capture long-range dependencies for accurate segmentation boundaries [47] and inherence of generalizability of models even with no pre-training strategies and applicable for low-data regime [108], [109], [163], Huang et al.”
REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs (2019, neutral): “The challenge proposal was accepted after assessing the compliance to good practices proposed in (Maier-Hein et al., 2018; Reinke et al., 2018).”
Artificial intelligence and machine learning for medical imaging: A technology review. (2021, neutral): “However, the results and rankings from competitions must be interpreted carefully when transferring the acquired knowledge into clinical applications [245,246].”
A large annotated medical image dataset for the development and evaluation of segmentation algorithms (2019, background): “Many key algorithmic enhancements in the field are validated on a relatively small number of samples [6] with only a few comparisons to existing approaches, thereby limiting our understanding of the generalizability of the proposed algorithms and our ability to differentiate novel from incremental a”
VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images (2020, methodology): “Inspired from (Maier-Hein et al., 2018) and (Menze et al., 2014), comparison and ranking of the participating algorithms was based on a scheme based on statistical significance.”
The Medical Segmentation Decathlon (2021, methodology): “To investigate ranking uncertainty and stability, bootstrapping methods were applied with 1,000 bootstrap samples as described in [10].”