Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge

By Hugo J. Kuijf, A. Casamitjana, D. Louis Collins, M. Dadar, Achilleas Georgiou, Mohsen Ghafoorian, Dakai Jin, A. Khademi, Jesse Knight, Hongwei Li and 34 more. IEEE Transactions on Medical Imaging, 2019

315 citations recorded by the source corpus.

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. The automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their methods on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge. Sixty T1 + FLAIR images from three MR scanners were released with the manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. The segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: 1) Dice similarity coefficient; 2) modified Hausdorff distance (95th percentile); 3) absolute log-transformed volume difference; 4) sensitivity for detecting individual lesions; and 5) F1-score for individual lesions. In addition, the methods were ranked on their inter-scanner robustness; 20 participants submitted their methods for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all the methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.

Subfields: Computer Vision and Pattern Recognition, Epidemiology, Neurology

View Paper: https://www.semanticscholar.org/paper/1e7ba4b8b2332a73b978b94b99ab356687bb2b67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index (2020, background): “However, often multiple metrics are of interest for evaluation such as volume errors or distance-based metrics [3], [30].”
Domain Adaptation for Medical Image Analysis: A Survey (2021, unknown): “Feature transformation strategy focuses on transforming source and target samples from their original feature spaces to a new shared feature representation space [47]–[49].”