Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

By Abdel Aziz Taha, Allan Hanbury. BMC Medical Imaging, 2015

2,743 citations recorded by the source corpus.

BackgroundMedical Image segmentation is an important image processing step. Comparing images to evaluate the quality of segmentation is an essential part of measuring progress in this research area. Some of the challenges in evaluating medical segmentation are: metric selection, the use in the literature of multiple definitions for certain metrics, inefficiency of the metric calculation implementations leading to difficulties with large volumes, and lack of support for fuzzy segmentation by existing metrics.ResultFirst we present an overview of 20 evaluation metrics selected based on a comprehensive literature review. For fuzzy segmentation, which shows the level of membership of each voxel to multiple classes, fuzzy definitions of all metrics are provided. We present a discussion about metric properties to provide a guide for selecting evaluation metrics. Finally, we propose an efficient evaluation tool implementing the 20 selected metrics. The tool is optimized to perform efficiently in terms of speed and required memory, also if the image size is extremely large as in the case of whole body MRI or CT volume segmentation. An implementation of this tool is available as an open source project.ConclusionWe propose an efficient evaluation tool for 3D medical image segmentation using 20 evaluation metrics and provide guidelines for selecting a subset of these metrics that is suitable for the data and the segmentation task.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4697b29af0d3c85049f5e338f62a28b3fec821e5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Non-local Deep Features for Salient Object Detection (2017, methodology): “Motivated by the signiﬁcant applications of Dice loss or IoU boundary loss in medical image segmentation [53, 40, 34], our proposed method approximates the penalty on boundary length of Eq.”
The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment (2021, methodology): “In the future, we plan to make additional comparative analyses between the Matthews correlation coefﬁcient and other rates, such as the Fowlkes-Mallows index [101], the prevalence threshold [102], and the Jaccard index [103], [104].”
The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification (2023, background): “Broadly speaking, it is always better to employ multiple statistics for results’ evaluation, in any scientific project [66–70].”
TractSeg ‐ Fast and accurate white matter tract segmentation (2018, neutral): “…matter tracts is virtual dissection : streamlines that correspond to anatomically well-defined tracts are manually extracted from a tractogram using combinations of inclusion and exclusion of regions-of-interest (ROIs) (Wang et al., 2007, Stieltjes et al., 2013, Thiebaut de Schotten et al., 2011).”
VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images (2020, methodology): “Putting aside the insufficiency of the Dice metric for evaluating segmentation performance (Taha & Hanbury, 2015), the metrics in the spine literature have a major short-coming: one-label shift, where the labels of the predicted mask are off by one label (cf. Fig.”
Dynamic Snake Convolution based on Topological Geometric Constraints for Tubular Structure Segmentation (2023, methodology): “Hausdorff Distance (HD) [26] is also widely used to describe the similarity between two sets of points, which is recommended to evaluate the thin tubular structures.”
REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs (2019, neutral): “ al., 2015). Although dierent by denition, these two metrics can be computed from each other, as they are dened as ratios of overlap between the predicted area and the manual reference annotation (Taha and Hanbury, 2015). Pixelwise sensitivity and specicity values have been also reported in (Al-Band”
Towards a guideline for evaluation metrics in medical image segmentation (2022, neutral): “Still, by including the precision, it also penalizes false positives, which is a common factor in highly class imbalanced datasets like MIS [10,11].”
Metrics Reloaded: Recommendations for image analysis validation (2022, background): “Generating and handling fuzzy reference data (for example, from multiple observers) is a topic of its own 31,32 and was decided to be out of scope for this work.”
clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation (2020, methodology): “For example, Dice and Jaccard rely on the average voxel-wise hit or miss prediction [46].”
Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks (2019, methodology): “The performance of automatic segmentation methods is usually evaluated by computing some common objective criteria such as the Dice similarity coefficient (DSC), mean boundary distance, volume difference or overlap, and Hausdorff Distance (HD) [5], [6].”
Why rankings of biomedical image analysis competitions should be interpreted with care (2018, background): “Be aware of metric-specific biases in favor of/against various properties [Taha and Hanbury 2015].”