Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge

By Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S. N. Berens, C. V. D. Bogaard, P. Cerello, Hao Chen, Qi-Yan Dou, M. Fantacci, B. Geurts and 21 more. Medical Image Anal., 2016

1,345 citations recorded by the source corpus.

HighlightsA novel objective evaluation framework for nodule detection algorithms using the largest publicly available LIDC‐IDRI data set.The impact of combining individual systems on the detection performance was investigated.The combination of classical candidate detectors and a combination of deep learning architectures generates excellent results, better than any individual system.Our observer study has shown that CAD detects nodules that were missed by expert readers.We released this set of additional nodules for further development of CAD systems. Graphical abstract Figure. No caption available. ABSTRACT Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC‐IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC‐IDRI data. We released this set of additional nodules for further development of CAD systems.

Subfields: Pulmonary and Respiratory Medicine, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7975bfe13445f81edddd6e101d7d469cd20cbd89

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Pulmonary and Respiratory Medicine

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Medical image data augmentation: techniques, comparisons and interpretations (2023, methodology): “Lung images have been provided from Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) database (Armato et al. 2011; McNitt-Gray et al. 2007; Wang et al. 2015) by the LUNA16 challenge (Setio et al. 2017).”
3D G-CNNs for Pulmonary Nodule Detection (2018, methodology): “5mm, allowed us to use the reference standard for nodule detection as used by the LUNA16 grand challenge (26) and performance metric as specified by the ANODE09 study (27).”
Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules (2017, background): “Nod-ules with diameter 3mm are excluded [10].”
DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning (2018, methodology): “We merged bounding-boxes that have more than 60% overlap by averaging their coordinates.(16)”
Metrics Reloaded: Recommendations for image analysis validation (2022, positive): “• Community preferences: While AP constitutes the undisputed standard metric for object detection and instance segmentation in the computer vision community, the FROC Score is often favoured in the clinical context due to its easier interpretability despite its lack of standardization (employed Fals”
fastMRI: An Open Dataset and Benchmarks for Accelerated MRI (2018, background): “which include a range of medical imaging modalities, are designed to test the limits of current methods in the tasks of segmentation, classi- cation, and detection. Datasets such as BraTS [24], LUNA [37], ChestX-ray [50], DeepLesion [55], and Camelyon [1] are widely used in research with clinical ap”
Automated Lung Nodule Detection and Classification Using Deep Learning Combined with Multiple Strategies (2019, background): “Dr. Silvestri and his research team have proposed proteomic classiﬁers to di ﬀ erentiate between small size benign and malignant nodules [3].”
Hybrid deep learning for detecting lung diseases from X-ray images (2020, positive): “[12] for the reduction of false positive rate in classifying the lung nodules.”
Fifty years of computer analysis in chest imaging: rule-based, machine learning, deep learning (2017, background): “The results, recently summarized by Setio et al. [50], are unambiguous: systems based on convnets perform substantially better than do classical machine learning approaches.”
From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge (2019, background): “challenges in recent years, in many medical imaging fields, for example: liver segmentation in CT (SLIVER07) [10], brain tumor segmentation in MRI (BRATS) [11], or lung nodule detection in CT (LUNA16) [12].”
Evaluate the Malignancy of Pulmonary Nodules Using the 3-D Deep Leaky Noisy-OR Network (2017, methodology): “A competition called LUng Nodule Analysis 2016 (LUNA16) [22] was held based on a selected subset of LIDC.”