From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge

By Péter Bándi, Oscar G. F. Geessink, Quirine F. Manson, M. V. van Dijk, M. Balkenhol, M. Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, K. Paeng, Aoxiao Zhong and 26 more. IEEE Transactions on Medical Imaging, 2019

709 citations recorded by the source corpus.

Automated detection of cancer metastases in lymph nodes has the potential to improve the assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 Conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images (WSIs) for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 WSIs. The evaluation metric used was a quadratic weighted Cohen’s kappa. We discuss the algorithmic details of the 10 best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre- and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy, and pre- and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to −0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targeted for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.

Subfields: Pulmonary and Respiratory Medicine, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/1b27b9cfe0ce17950b6ea72f9ef8cf5a7459bccd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Pulmonary and Respiratory Medicine

Constantinos S. Pattichis

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, background): “Camelyon17-WILDS (Bandi et al., 2018; Koh et al., 2020) is an image dataset of metastasized breast cancer tissue samples collected from different hospitals.”
Artificial intelligence for digital and computational pathology (2023, background): “The resulting embedding is then passed to a predictor for predicting the corresponding patch label.”
Adaptive Weighting Multi-Field-Of-View CNN for Semantic Segmentation in Pathology (2019, methodology): “Many methods have been proposed for pattern recognition tasks in pathology: Patient-level and WSI-level pNstage estimation (CAMELYON 2016 and 2017) [4][5], and segmentation of tumors.”
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images (2021, background): “Though unable to model complex interactions between instances, MIL is able to solve many needle-in-a-haystack problems in pathology, as the classiﬁcation of normal tissue vs micro-metastases depends on discriminating only binary instance-level visual concepts [3, 35].”
Deep Learning for Whole Slide Image Analysis: An Overview (2019, methodology): “Most successful approaches also employ hard negative mining, an iterative process whereby false positives are added to the training data set for further training (14, 23, 24, 36).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, positive): “The Camelyon17-wilds dataset is adapted from whole-slide images (WSIs) of breast cancer metastases in lymph nodes sections, obtained from the CAMELYON17 challenge (Bandi et al., 2018).”
Deep neural network models for computational histopathology: A survey (2019, background): “The expansion of the breast cancer metastases dataset, CAMELYON16, to CAMELYON17 provides a good illustration of how much larger datasets are needed to assess an algorithm in a more meaningful clinical context (Litjens et al., 2018); in CAMELYON16 399 WSIs from 2 centres were made available but slid”
Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models (2022, methodology): “training distribution di ers from the test distribution, we evaluate on three image classi cation datasets in WILDS [35]: Camelyon17 [36], FMoW [37], and iWildCAM [38].”
Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images (2018, background): “For more than a century, the interpretation and determination of aberrant phenotypes in these stained tissue specimens has been accomplished by subjective human interpretation [7–9].”
AI in Medical Imaging Informatics: Current Challenges and Future Directions (2020, positive): “The CAMELYON challenges target methods for algorithmic detection and classification of breast cancer metastases in H&E whole slide lymph node sections [171].”
Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis (2019, background): “Anatomic pathology has the ability to reveal the inherent phenotypic intratumoral heterogeneity of cancer, and has been an important tool in cancer prognosis for the past century [33]– [36].”
Exploring Visual Prompts for Adapting Large-Scale Models (2022, methodology): “We also measure robustness to distribution shift, i.e., training distribution differs from the test distribution, by evaluating on three image classiﬁcation datasets in WILDS [23]: Camelyon17 [1], FMoW [8], and iWildCAM [2].”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[8] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al.”