Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge

By Keelin Murphy, B. van Ginneken, J. Reinhardt, S. Kabus, K. Ding, Xiang Deng, K. Cao, K. Du, Gary E. Christensen, Vincent Garcia and 44 more. IEEE Transactions on Medical Imaging, 2011

492 citations recorded by the source corpus.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intra patient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the con figuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Subfields: Biomedical Engineering, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/8b8d569cbd9129d9e25843e7156b7d8244eba30e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Gland segmentation in colon histology images: The glas challenge contest (2016, background): “Furthermore, to ensure blindness of evaluation, the ground truth and histologic grade of each image in the test parts were not released to the participants.”
Tracking deformable objects with point clouds (2013, background): “[13] give an assessment of modern approaches to this problem.”
Survey of Non-Rigid Registration Tools in Medicine (2017, methodology): “Most of the reviewed tools were established for brain image registrations, but many of the registration algorithms used in these tools were applied successfully to other registration tasks, for example, the registration of thoracic CT [59].”
MRF-Based Deformable Registration and Ventilation Estimation of Lung CT (2013, methodology): “The results are also compared to the state-of-the-art continuous optimization framework gsyn [12], which performed best in a recent lung registration challenge [5], and two methods from the literature, which have been evaluated using the same dataset.”
Image Similarity and Tissue Overlaps as Surrogates for Image Registration Accuracy: Widely Used but Unreliable (2011, methodology): “…evaluation data set for longitudinal (within-subject) thoracic CT registration has recently become available from the EMPIRE10 Challenge [26], [27], which uses as evaluation criteria the alignment of lung boundaries , major fissures, and up to 100 landmark pairs, as well as…”
A survey of medical image registration - under review (2016, unknown): “Among the few exceptions of evaluation of nonlinear registration accuracy that do employ clinical data is the EMPIRE10 challenge mentioned earlier ( Murphy et al., 2011a ).”
An Artificial Agent for Robust Image Registration (2016, background): “While medical image registration has been an active research area for more than two decades (Markelj et al. 2012; Murphy et al. 2011), fully automatic and robust 3-D registration remains a challenging task that requires manual intervention for possible corrections.”
Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge (2019, methodology): “This has been successfully applied to various tasks, such as liver segmentation [7], image registration [8], coronary calcium scoring [9], or gland segmentation in histology images [10].”
ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI (2016, methodology): “To this end, we planned, organized, (Styner et al., 2008) in 161 MRI brain data; complete lungs (Murphy, 2011) or their ves-162 sels (Rudyanto et al., 2014) in computed tomography scans; 4D 163 ventricle extraction (Petitjean et al., 2015) as well as myocardial 164 tracking and deformation…”
Unsupervised End-to-end Learning for Deformable Medical Image Registration (2017, methodology): “The ROI segmentation modules are trained only by original data of MICCAI 2007 Grand Challenge [48], and the registration modules are trained in a completely unsupervised manner.”
Whole-brain imaging with single-cell resolution using chemical cocktails and computational analysis. (2014, unknown): “For image registration, we used advanced normalization tools (ANTs), which have proved successful in resolving a number of image registration issues (Murphy et al., 2011).”
Deformable Medical Image Registration: A Survey (2013, methodology): “The development of dedicated methods for reference standard construction [454] and the organization of registration 2 http://www.mindboggle.info/papers/evaluation NeuroImage2009/data/ challenges [455] create the necessary conditions for objective comparison of registration methods.”
Lung Segmentation in Chest Radiographs Using Anatomical Atlases With Nonrigid Registration (2013, unknown): “The fuzzy-membership is estimated with Kohonen networks [7].”