Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge

By Geert J. S. Litjens, R. Toth, W. J. M. Ven, C. Hoeks, Sjoerd Kerkstra, B. van Ginneken, Graham Vincent, Gwenaël Guillard, N. Birbeck, Jindang Zhang and 18 more. 2013

882 citations recorded by the source corpus.

Prostate MRI image segmentation has been an area of intense research due to the increased use of MRI as a modality for the clinical workup of prostate cancer. Segmentation is useful for various tasks, e.g. to accurately localize prostate boundaries for radiotherapy or to initialize multi-modal registration algorithms. In the past, it has been difficult for research groups to evaluate prostate segmentation algorithms on multi-center, multi-vendor and multi-protocol data. Especially because we are dealing with MR images, image appearance, resolution and the presence of artifacts are affected by differences in scanners and/or protocols, which in turn can have a large influence on algorithm accuracy. The Prostate MR Image Segmentation (PROMISE12) challenge was setup to allow a fair and meaningful comparison of segmentation methods on the basis of performance and robustness. In this work we will discuss the initial results of the online PROMISE12 challenge, and the results obtained in the live challenge workshop hosted by the MICCAI2012 conference. In the challenge, 100 prostate MR cases from 4 different centers were included, with differences in scanner manufacturer, field strength and protocol. A total of 11 teams from academic research groups and industry participated. Algorithms showed a wide variety in methods and implementation, including active appearance models, atlas registration and level sets. Evaluation was performed using boundary and volume based metrics which were combined into a single score relating the metrics to human expert performance. The winners of the challenge where the algorithms by teams Imorphics and ScrAutoProstate, with scores of 85.72 and 84.29 overall. Both algorithms where significantly better than all other algorithms in the challenge (p < 0.05) and had an efficient implementation with a run time of 8 minutes and 3 second per case respectively. Overall, active appearance model based approaches seemed to outperform other approaches like multi-atlas registration, both on accuracy and computation time. Although average algorithm performance was good to excellent and the Imorphics algorithm outperformed the second observer on average, we showed that algorithm combination might lead to further improvement, indicating that optimal performance for prostate segmentation is not yet obtained. All results are available online at http://promise12.grand-challenge.org/.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation (2016, methodology): “We trained our method on 50 MRI volumes, and the relative manual ground truth annotation, obtained from the ”PROMISE2012” challenge dataset [10].”
Iterative Interaction Training for Segmentation Editing Networks (2018, neutral): “Recent studies showed that CNN-based approaches were able to achieve inter-and intra-expert performance in certain segmentation tasks, for example prostate segmentation in Magnetic Resonance Images (MRIs) as shown in [8,4].”
NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “On Promise12, NAS-Unet significantly outperforms the baseline methods.”
Federated learning for medical image analysis: A survey (2023, background): “2) Abdomen: PROMISE12: The MICCAI 2012 Prostate MR Image Segmentation challenge dataset (PROMISE12) [135] is a publicly available dataset for the evaluation of prostate MRI segmentation methods.”
ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI (2016, methodology): “103 While all approaches make an e ﬀ ort to quantify segmenta-104 tion accuracies, most lack detailed descriptions of the employed 105 Table 1: Listing of publications describing non-chronic stroke lesion segmentation in MRI with evaluation on human image data since Rekik et al. (2012).”
Computer Aided-Diagnosis of Prostate Cancer on Multiparametric MRI: A Technical Review of Current Research (2014, methodology): “In the field of prostate segmentation for MRI, we are pleased to see the PROMISE12 challenge has been successfully organized [48].”
Cancer Diagnosis Using Deep Learning: A Bibliographic Review (2019, methodology): “Eight studies [12,59,77,87,96,98,103,105] used CT Slices, three of which used data from PROMISE [75], and LIDC [166].”
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation (2022, methodology): “It first learns a separate segmentation model for each view on labeled data, and then the predictions of the models on unlabeled data are [22], PROMISE [182] Task-level consistency 1. https://cremi.org/ the views.”
MS-Net: Multi-Site Network for Improving Prostate Segmentation With Heterogeneous MRI Data (2020, methodology): “We construct an external testing set of 20 samples from Promise12 dataset [55].”
Segment Anything in Medical Images (2023, background): “, T1, T2, ADC, FLAIR) [24][10][6][41][5][8][29][42], abdomen tumor, COVID-19 infection, gallbladder, head and neck tumor, liver, pancreas, pleural effusion, and stomach segmentation in CT images [34][19][7][33][32][42][4][28][27][12], breast tumor, liver, and vessel segmentation in ultrasound images”
Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation (2019, methodology): “(A) For prostate segmentation, samples are marked by their domain: PROSTATEx (source domain), and the multi-device multi-institutional PROMISE12 dataset (target domain).”
Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images (2017, methodology): “Note that all evaluation metrics including both boundary and volume metrics were calculated in 3D not only for the entire prostate segmentation but also specially for the apex and base parts of the prostate, which are the most difficult yet important parts in clinical practice (Litjens et al. 2014).”
Medical Image Segmentation Using Deep Learning: A Survey (2020, neutral): “Prostate PROMISE12 [160] https://promise12.”