Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge

By Esther E. Bron, Marion Smits, W. Flier, Hugo Vrenken, Frederik Barkhof, Philip Scheltens, J. Papma, Rebecca M. E. Steketee, C. Orellana, R. Meijboom and 17 more. NeuroImage, 2015

349 citations recorded by the source corpus.

Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer’s disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with in total 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimer’s Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Psychiatry and Mental health

View Paper: https://www.semanticscholar.org/paper/57da7e969b459d689277c0c3643d9bf957f454fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Psychiatry and Mental health

Citations in the corpus

Domain Adaptation for Medical Image Analysis: A Survey (2021, methodology): “Experiments on the ADNI [49], AIBL [50] and CADDementia challenge datasets [51] show that the DA model can achieve better results than methods that only use data from either the source or target domain.”
A review on neuroimaging‐based classification studies and associated feature extraction methods for Alzheimer's disease and its prodromal stages (2017, positive): “For example, the purpose of CADDementia 2015 challenge (Bron et al., 2015) was to measure the generalizability of structural MRI-based classification studies on unseen subjects, where the best performing study yielded an area under the receiver-operating-characteristic curve of 78.”
Predicting brain-age from multimodal imaging data captures cognitive impairment (2016, positive): “Individual brain-based prediction and classification may give rise to brain imaging-based biomarkers that could aid clinical diagnostics, for instance, by predicting an individual’s risk of developing dementia based on their brain (Bron et al., 2015).”
Alzheimer's disease diagnostics by adaptation of 3D convolutional network (2016, methodology): “Voxel-wise, cortical thickness, and hippocampus shape-volume features of the sMRI are used to diagnose the AD [3].”
Deep Learning Approach for Early Detection of Alzheimer’s Disease (2021, methodology): “Computer-Aided System (CAD) is used for accurate and early AD detection to avoid AD patients ’ high care costs, which are expected to rise dramatically [12].”